Tokyo's Nikkei average closed down 0.17 percent on Tuesday as growing worries over the domestic economic outlook hit banks and insurers, offsetting gains in shipping firms and others with bright earnings prospects. Technology exporters such as Sony Corp were also sold as investors remained cautious about the US economic outlook and about how the dollar's recent weakness would affect companies' earnings.
The Nikkei shed 18.96 points to 10,964.87 after swinging between gains and losses.
The broader TOPIX index fell 0.22 percent to 1,100.33.
Analysts said investors, already nervous about Japan's macroeconomic picture, were bracing for a slew of economic data.
Japan's machinery orders data for September, regarded as a leading gauge of capital spending, will be released on Thursday, followed by third-quarter gross domestic product (GDP) figures and revised September industrial output, both on Friday.
"Some are saying GDP will be weaker than expected so we have to see that (GDP). The market is likely to go down a bit by Friday, though, because of the uncertainty," said Michael Coates, director of equity sales at KBC Financial Products.
The US Federal Reserve's interest rate decision on Wednesday and its comments on the US economy are also keenly awaited, others said.
Suggesting investor caution, trade volume declined to 1.07 billion shares, down from Monday's 1.08 billion and the lowest total since October 18.
Decliners slightly outnumbered gainers 717 to 697.
Banks extended falls into a second session, with Mizuho Financial Group Inc falling 0.95 percent to 418,000 yen and Mitsubishi Tokyo Financial Group losing 0.97 percent to 922,000 yen.
Japan's biggest non-life insurer Millea Holdings Inc dropped 1.32 percent to 1.49 million yen.
Consumer electronics giant Sony shed 0.8 percent to 3,740 yen while Canon Inc, Japan's biggest office machine maker which generates three quarters of sales overseas, dropped 1.12 percent to 5,320 yen.
The dollar, which hit a seven-month low of 105.28 yen on Monday, was trading at around 105.65 yen on Tuesday. A higher yen eats into exporters' profits when they are repatriated and makes Japanese products more expensive abroad.
Mitsui O.S.K. Lines, Japan's second-biggest shipping company, rose 2.27 percent to 631 yen.
The daily Nihon Keizai said Mitsui was likely to post 155 billion yen ($1.47 billion) in group operating profit for the year to March, higher than the company's revised profit forecast of 144 billion yen in August.