The pair, now facing espionage charges, were detained after Canadian police picked up Huawei Technologies Co Chief Financial Officer Meng Wanzhou in Vancouver last December on a U.S. arrest warrant.
Trudeau's government has pressed the United States to do more for the men. Pompeo said President Donald Trump had raised the case directly with Chinese President Xi Jinping during a meeting in late June.
"Please do note that our team is focused on helping those two Canadians be released," Pompeo said at the beginning of a meeting in Ottawa with Trudeau and Canadian Foreign Minister Chrystia Freeland.
"It's wrong that they are being held."
Pompeo later said the United States had made clear from the start that the detention of the two men had been inappropriate.
"We have engaged in other diplomatic activity, trying to help make the case for the release of these two," he told a news conference at the end of the talks.
"We will continue to do that until such time as they are home and have been returned to their families."
Pompeo was visiting Ottawa for the day ahead of a weekend summit of the Group of Seven industrialized nations in France where relations with China are set to be discussed.
The United States is locked in a trade war with China that is upsetting world markets.
Asked whether Washington saw Meng as a bargaining chip in the dispute, Pompeo replied "No".
Freeland reiterated Canada's position that it would not ask the United States to drop its request for Meng's extradition. She faces U.S. charges of conspiring to defraud global banks about Huawei's relationship with a company operating in Iran.
Speaking in Beijing, Chinese foreign ministry spokesman Geng Shuang reiterated that the two Canadians were both suspected of national security crimes and that China's legal authorities handled such cases independently.
The cases had nothing to do with the United States and they had no right to "make thoughtless remarks" about it, Geng told reporters.
Trudeau said on Wednesday his government had no intention of backing down in the dispute with China and would defend Canada's interests.
In response, the Chinese embassy in Ottawa on Thursday blamed Canada for "gross difficulties" in bilateral ties and demanded Meng's release.
Freeland and Pompeo also discussed the crisis in Venezuela, where President Nicolas Maduro is resisting international pressure to step down and allow elections to take place.
Freeland said she would travel to Cuba next week to discuss possible ways of ending the crisis.
Pompeo's visit is the latest indication of how bilateral ties have improved since hitting a low point in June 2018 when Trump called the Canadian leader weak and dishonest.
The United States, Mexico and Canada signed a new continental trade pact in November. But the U.S.-Mexico-Canada (USMCA) treaty still needs to be ratified by Canada and the United States, where some Democrats have expressed concern about the pact's provisions on labor.
"I want to reiterate we need to get the USMCA wrapped up and we need our parliaments both to ratify them. It will be good for the people all across North America," Pompeo said at the start of the meeting with Trudeau.