The two Canadians -- former diplomat Michael Kovrig and businessman Michael Spavor -- were detained in December and accused of espionage.
The detentions came nine days after Canada arrested Huawei executive Meng Wanzhou on a US warrant. While no official link was made, the arrest of the two men was widely viewed as retaliation.
"Please note that our team is focused on helping those two Canadians be released," Pompeo told Trudeau. "We're working on it diligently."
"It's wrong that they are being held," he added.
Pompeo was in Ottawa to discuss trade, China and the upcoming G7 summit with Trudeau and Foreign Minister Chrystia Freeland.
He also met with Canadian business executives -- including Canada's largest agricultural exporter Richardson International, which saw its canola shipments to China blocked this year.
Later, Pompeo told a joint press conference with Freeland: "The United States stands with Canada in the face of China's arbitrary and unacceptable detention of Canadian citizens."
He noted that US President Donald Trump has raised their detention directly with Chinese President Xi Jinping and "made unambiguous America's concern about this inappropriate behaviour."
On the eve of the Pompeo visit, Trudeau vowed in a speech that he would "always defend Canadians and Canadian interests" and not "back down" in deepening diplomatic and trade disputes with China.
Beijing quickly shot back, with Chinese foreign ministry spokesman Geng Shuang blaming Ottawa for the row and renewing Beijing's demands that Canada immediately release Meng.
The two countries have been locked in a feud since Meng's arrest during a flight stopover in Vancouver.
The US is seeking her extradition to face fraud charges for allegedly violating Iran sanctions and lying about it to US banks -- accusations that her lawyers dispute.
Pompeo commented that China "wants to talk about these two (Meng and the Canadian pair) as if they're equivalent and morally similar, which they fundamentally are not."
"The arbitrary detention of two Canadian citizens in China is a fundamentally different matter than the Canadian decision to apply the rule of law that's consistent with the way decent nations work."
"No," America's top diplomat said when asked if Washington might use Meng as a bargaining chip in the US trade war with China.
Trump had said in an interview with Reuters last December that he might intervene in the case if it would help secure a trade deal.
Meng's lawyers have pointed to the remarks to suggest that the charges against their client are politically motivated.
"It is a legal process by the United States Department of Justice designed to bring someone who we believe we have sufficient information to bring back to the United States under an (extradition) agreement between the United States and Canada," Pompeo insisted.
"Extradition is a criminal justice matter. It is not a political matter," echoed Freeland. "The case of Ms. Meng is currently before the Canadian courts, as it ought to be."
Meng is currently out on bail in Vancouver awaiting an extradition hearing scheduled to start in January.
On Wednesday, her lawyers alleged in court documents that she was unlawfully detained and questioned by Canadian border agents at the Vancouver airport last year.