The benchmark Nikkei 225 index slipped 0.69 percent, or 150.65 points, to 21,535.25, while the broader Topix index fell 0.48 percent, or 7.57 points, to 1,568.74.

The yen's relative rise made Tokyo players wary, offsetting positive factors led by new record highs in New York, analysts said.

The dollar stood at 107.95 yen, slightly up from 107.87 yen in New York on Monday but still down from 108.34 yen on Friday, before the long weekend in Japan.

"The yen is still at acceptable levels but a further gain wouldn't be good," said Daiwa Securities chief technical analyst Eiji Kinouchi.

Trading was sluggish as investors stayed on the sidelines ahead of the nation's upper house election on Sunday, with Prime Minister Shinzo Abe's ruling coalition forecast to secure a majority.

"A victory by the ruling coalition will be positive for the market," Kinouchi told AFP.

Among major shares, exporters slumped.

Sony dropped 3.02 percent to 5,866 yen, while Nintendo edged down 0.09 percent to 41,610 yen.

Auto shares were mixed. Toyota rose 1.30 percent to 7,000 yen but Nissan lost 0.16 percent to 774.1 yen, with Honda down 0.36 percent at 2,836.5 yen.

Mitsubishi Heavy Industries rose 0.48 percent to 4,768 yen following news reports that South Korean plaintiffs who won a wartime labour compensation case against the firm would proceed with the sale of its assets in the country.