England take on the Netherlands in the Nations League semi-final on Thursday three months after a 5-1 Euro 2020 qualifying win in Podgorica that was overshadowed by monkey chanting from some home supporters, which sparked a call from Manchester City striker Raheem Sterling for stadium closures in the event of racist abuse.
But in an interview with AFP, FIFA Deputy Secretary General Boban said he does not believe suspending matches for abuse dished out by a small number of spectators is the right move.
"If 25 percent of the stadium have a really discriminatory behaviour, yes ... but if for five people, for three guys, we have to stop the game and empty the stadium, I'm not in favour, I'm in favour of these people being put in jail," Boban said.
"Why should we be defeated by few idiots? Now because of three drunk fascists, we should stop the game? They should be taken out (of the stadium), then prosecuted immediately.
"But we need the help from authorities; FIFA doesn't control the police."
Montenegro will have to play their Euro 2020 qualifier against Kosovo on June 7 behind closed doors after being hit in April with a one-match stadium ban, which they had unsuccessfully appealed despite the furore surrounding the abuse.
In April, England and Tottenham defender Danny Rose insisted the punishment dished out to Montenegro by UEFA "was not harsh enough" and said he hoped he would "never have to play there again".
"I don't think it is a harsh enough punishment for someone to learn from in the future, just a one-game ban and 20,000 euros -- it's a bit shocking but there is not much I can do now," Rose said.
Former AC Milan and Dinamo Zagreb midfielder Boban also told AFP he was disappointed that the 2022 World Cup in Qatar would not be a 48-team competition after FIFA shelved the proposal last month.
World football's governing body dropped the idea due to problems arising from the likely need for Qatar's neighbours to host matches, which would require those states to put aside a two-year blockade of the World Cup hosts.
"I'm sad that we didn't achieve it but we tried, as our president said, we had to give it a try ... but it was a short time, with a lot of logistical and other complications, so it's better we dropped it," he said.