Migrant camps were "very prone" to spreading the disease, IOM Director General Antonio Vitorino told reporters in an online briefing.
"There are thousands of stranded migrants all over the world," he said.
"Lots of migrants that were on the move, some of them wanted to return to their country of origin, precisely because of the pandemic," he said, adding that others had become "stranded" on their migratory routes.
"They are blocked in the border areas in very difficult conditions without access to minimal care, especially health screening," he said.
"This is a source of enormous concern."
Vitorino said the IOM was asking governments to allow aid workers in to access large groups congregating near borders.
The former Portuguese defence minister said overcrowded conditions among migrants stranded at borders are so bad that "social distancing is unthinkable and access to water and sanitation is quite a challenge.
"We cannot ask people to do what is impossible," he said.
"If the disease spreads in the camps it will have a major impact."
The IOM manages around 1,100 camps worldwide and assisted roughly 2.4 million displaced people in 2019.
Some 220 cases of COVID-19 illness have been reported in IOM-run migrant camps in mainland Greece and been treated in the Greek healthcare system.
Vitorino said he was particularly concerned about what might happen if the virus took hold in Cox's Bazaar in Bangladesh, where more than a million Rohingya refugees have fled to from neighbouring Myanmar.
He said the crisis had caused a decrease in attempted crossings of the Mediterranean Sea between Africa and Europe.
But the lockdown restrictions have only created "a backlog of people waiting" - stranded migrants who, once movement restrictions start to lift, will head for Europe again.
Vitorino said the criminal networks of smugglers and traffickers were "ready to start working immediately" after the crisis subsides.
The IOM was also worried about migrants being held in detention and those in "extremely prone" urban slums, particularly in South America.