The two companies have combined annual sales of more than 12 trillion yen ($155 billion) and the move would create a Japanese industrial behemoth and one of the world's biggest infrastructure firms.
Hitachi's president was reported as saying his company and Mitsubishi Heavy Industries had agreed to start merger talks.
"We will negotiate a merger from now," Hitachi president Hiroaki Nakanishi told reporters in Yokohama, near Tokyo, according to Kyodo news agency.
Kyodo and other media quoted sources saying the two firms aimed to integrate infrastructure businesses such as railways and power stations, in early 2013.
The news boosted shares of the two companies. Hitachi was up 3.23 percent at 478 yen in late morning trade and Mitsubishi Heavy Industries was up 4.58 percent at 365 yen.
"With opportunities for the social infrastructure business expected to expand especially in emerging countries like China and India, the two Japanese companies on their own do not have sufficient management resources to compete globally," said Yukihiko Shimada, an analyst at SMBC Nikko Securities.
"If they can complement each other in this aspect and effectively merge their operations, there will be a positive synergy," he said.
The companies declined to confirm the reports.
Hitachi said in a statement that the merger talks were not "what we have decided or announced," while Mitsubishi Heavy said it had not decided on the move or planned to do so.
Analysts say further consolidation is needed in a Japanese corporate space that has too many companies making the same products compared with the likes of South Korea and industrial champions such as Samsung.
Earlier this year Japan's biggest steelmaker Nippon Steel and third-ranked rival Sumitomo Metal Industries said they were working towards a merger that would create the world's second-largest steel firm by 2012.
Mitsushige Akino, chief fund manager at Ichiyoshi Investment Management, said "there is a very strong sense of crisis" between Hitachi and Mitsubishi.
"It's positive that these talks are happening when the companies are on the brink of losing their competitiveness," he said.
The continuing crisis at the Fukushima nuclear plant has raised uncertainty over the future of their atomic power businesses, while the strength of the yen has made it more expensive for exporters to produce their goods domestically.
The yen had been hovering near its highest level since World War II in the past weeks, prompting Japanese authorities on Thursday to step into the market to weaken it.
Hitachi, Japan's largest maker of electrical machinery, said last week its net profit plunged 96.6 percent to 2.9 billion yen ($37 million) in the April-June quarter, citing the impact of the March 11 earthquake and tsunami.
It has been facing pressure to curb some of its unprofitable lines and focus more on core businesses.
It said Wednesday that it was considering shifting all television production to foreign outsourcing firms by March as part of a broad strategy to increase profitability.
Mitsubishi Heavy is Japan's largest heavy machinery manufacturer, with product lines ranging from ships, nuclear power plants, aerospace and engines. It is also Japan's largest military contractor.
It was to announce its quarterly earnings later Thursday.
Copyright AFP (Agence France-Presse), 2011