The former India captain is an honorary lieutenant colonel in India's Parachute Regiment and its dagger badge was printed on his wicketkeeper's gloves for India's win over South Africa on Wednesday.
International Cricket Council spokeswoman Claire Furlong said the Board of Control for Cricket in India has been ordered to remove the badge for India's game against Australia on Sunday and the rest of the tournament.
"It is against the regulations and we have requested it is removed," Furlong told Indian media. Only manufacturer's logos are allowed on gloves.
She added that for a first breach, the ICC had only requested the badge be removed, with no further punishment.
Dhoni's gesture gained widespread attention television and social media attention in India during Wednesday's game in Southampton. The Indian team made no immediate comment. But it has in the past signalled respect for the country's military.
In March, the team wore camouflage-style caps during a one-day game against Australia as a tribute after 40 paramilitaries were killed by a suicide bomber in Kashmir. Dhoni provided the caps.