Anwar and Mahathir are now locked in a power struggle, reviving a rivalry that has loomed over politics in the Southeast Asian nation for two decades. The pair had reconciled and joined forces before polls in 2018 to oust a corruption-plagued government that had ruled the country for six decades.
The king appoints the prime minister and had interviewed the country's MPs to work out who they backed, but Mahathir, who was appointed interim leader following his resignation, said no candidate with enough support emerged.
A candidate must have the backing of at least 112 MPs to become premier.
The king "says that the right forum will be parliament," said Mahathir on Thursday, following a morning meeting with the monarch. He added the legislature would sit on Monday to determine who has sufficient backing to become premier.
"However if the (parliament) fails to find a person with a majority, then we will have to go for a snap election," Mahathir said.
But the "Pact of Hope" alliance - the former ruling coalition, which is now backing Anwar to become premier - criticised the move.
After an emergency meeting, the alliance said that the call for a special parliament session to vote on the next premier was a "challenge to the rights and powers" of the king.
Mahathir - at 94, the world's oldest leader - initially appeared to have strong support to remain as premier and had announced he wanted to form a government of national unity, and was willing to return as leader.
However his idea of a unity government was rejected by leaders across the political spectrum, with some expressing concerns that it would hand him too much power.
He changed tack Thursday with a surprise announcement that his Bersatu party might nominate as a candidate his ally Muhyiddin Yassin, who was interior minister until the government collapsed. "If everybody chooses him, I'm okay," said Mahathir.
Bersatu member Redzuan Yusof said the party would "100 percent" back Muhyiddin as prime minister, news portal Malaysiakini reported.