The Southeast Asian country is the world's second biggest palm oil producer after Indonesia and would be hit hard by the EU's plan to cut its use in biofuels by 2030.
"Malaysia will file a WTO complaint... hopefully by November," Teresa Kok, the minister responsible for the edible vegetable oil, told AFP.
She said the attorney-general's office was working on the complaint and Malaysia hoped to cooperate with Indonesia in the case.
Palm oil is a major ingredient in a wide range of products from food to cosmetics but it has long been controversial as environmentalists say it drives deforestation, with huge swathes of rainforest logged in recent decades to make way for plantations.
Its use in food and cosmetics has already dropped in Europe, partly due to pressure from green groups on major corporations, but has been increasing in biofuels.
Prime Minister Mahathir Mohamad in March slammed the EU move to curb palm oil in biofuels, saying it risks opening up a trade war with Malaysia.
He has also warned of retaliation by buying new fighter jets from China instead of France's Rafale jet or the Eurofighter Typhoon.
Last week Indonesian President Joko Widodo promised to fight the EU's plans, telling Bloomberg News that "palm oil is a strategic commodity for Indonesia".