Sultan Haitham discussed the "close ties" between Oman and the United States with Pompeo, the official Oman News Agency said.
Experts say Oman's new ruler, aged 65, is likely to continue the foreign policies of his cousin's five-decade reign.
An ally to Western countries including the United States, Qaboos cultivated Oman's status as a neutral actor, maintaining warm ties with Washington's arch-rival Iran.
Pompeo arrived in Oman from Riyadh, where he reassured close Western ally Crown Prince Mohammed bin Salman that "the United States stands with Saudi Arabia in the face of... the continuing threat posed by the Iranian regime."
Last year the United States boosted its military presence in Saudi Arabia following a series of attacks in the Gulf that Washington and Riyadh blamed on Iran.
Oman has often acted as a mediator between Iran and its regional enemies. It played a key role in facilitating talks involving the United States that led to the 2015 Iranian nuclear deal.
But tensions in the Gulf spiked after US President Donald Trump's administration unilaterally withdrew from the agreement in 2018 and began reimposing sanctions on the Islamic republic.
Iran retaliated by scaling back some of its nuclear commitments.
While in Saudi, Pompeo and the Crown Prince also discussed a resurgence in violence between Yemen's Huthi rebels and the Saudi-led coalition in the country. They "agreed on their support for UN Special Envoy Martin Griffiths' efforts to advance the political process there", the State Department said.
Before the Gulf, Pompeo visited Senegal, Angola and Ethiopia on his first African tour.