Yasuo Ichikawa told Japanese media just before he was appointed on Friday: "I am an amateur concerning security", in comments that the opposition Liberal-Democratic Party said proved he was not qualified for the job.
"For that comment alone he deserves to be discharged from his ministerial post," said LDP policy chief Shigeru Ishiba, a former defence minister.
He said that the wisdom of new New Prime Minister Yoshihiko Noda in appointing Ichikawa was also "put into question".
Noda, Japan's sixth new leader in five years, appointed his cabinet on Friday, featuring untested talent in key posts including the finance and foreign ministers.
Ichikawa, 69, who served in the farm ministry for 25 years before entering politics, said his comment had been misinterpreted.
"I meant to say that most of the people are amateurs and it is important to pursue security policies from the people's viewpoint," he said late Friday.
But the controversy refused to go away on Saturday with LDP policy expert Ichita Yamamoto joining calls for Ichikawa to quit as soon as possible.
"We feel very anxious leaving Japan's national defence to a person with such an attitude," Yamamoto said.
The ruling Democratic Party of Japan, often labelled centre-left, have been at odds with the United States over a huge US military presence in Okinawa since it ended the LDP's long domination of Japanese politics in 2009.
Noda's two predecessors have failed to resolve the issue with the key ally due to Okinawa islanders' resistance to the planned transfer of a US marine corps air station from a growing urban area to a scenic stretch of shore.