The benchmark Nikkei 225 index ended up 0.20 percent or 42.37 points at 21,685.90 in early trade, while the broader Topix index closed down 0.15 percent or 2.32 points at 1,576.31.

"Early gains were partly offset by profit-taking sales... with a lull in the trend of a cheaper yen against the dollar also weighing on the market," Okasan Online Securities said in a commentary.

Profit-taking came ahead of a long weekend in Japan, where markets will be closed on Monday for a public holiday.

Investors also remained cautious about the possible negative impact from US-China trade frictions and from the slowing Chinese economy, analysts said.

The latest Chinese data "shows a slowing pace of improvement", Resona Bank said in a note.

The dollar fetched 108.34 yen in early Asian trade, against 108.45 yen in New York.

In Tokyo, Fast Retailing, operator of the casual wear chain Uniqlo, gained 3.19 percent to 69,810 yen after reporting third-quarter operating profit largely in line with market expectations.

Tech firms were mixed, with Sony ending up 1.25 percent at 6,049 yen while industrial robot maker Fanuc closed down 1.98 percent at 19,215 yen.