Philanthropist Yu Panglin, who owns the mansion, which became a rundown 'love hotel', said he was planning to sell the property for HK$180 million ($23 million) after talks with the government for the museum collapsed last year.
"I'm no longer considering (the museum plan) since the government is not supportive," the 90-year-old billionaire told Hong Kong's Chinese-language Singtao Daily in an interview.
"I'm running out of patience, I don't want to wait anymore," he said, adding that the hotel owner had failed to pay rent for two years due to poor business, with the property in urgent need of refurbishment.
Lee's legions of fans have long-demanded a significant hometown tribute to the Chinese-American star, who died in 1973 at the age of 32 after helping to bring martial arts to the mainstream with classic kung fu films such as "Fists of Fury" (1971) and "Enter the Dragon" (1973).
However they were left disappointed after the Hong Kong government shelved the museum plan in June last year, saying it failed to reach a consensus with Yu following two years of negotiations, without giving details.
Yu told Singtao Daily the government rejected his proposal to expand the mansion -- Lee's last residence -- by adding three floors to include a cinema, library and martial arts training centre, which were his conditions to donate the property.
The 5,000 square-foot (460 square-metre) two-storey house in Hong Kong's upscale residential district of Kowloon Tong was turned into a short-stay hotel with rooms rented for as little as US$25 an hour, usually to amorous couples.
Yu could not be reached Wednesday for comment but Lee's fans urged the billionaire to reconsider his plan.
"If the house is sold and re-developed or demolished later, I think the Hong Kong people and Bruce Lee fans worldwide will be very disappointed," Wong Yiu-keung, chairman of the Bruce Lee Fan Club in Hong Kong, told AFP.
"We hope the government can try to persuade Yu again," he added.
The star died in Hong Kong after a severe reaction to medication. His fans have criticised the lack of a larger memorial to their hero in his hometown, except for a statue on the waterfront Avenue of Stars.
Authorities said in October last year a gallery would be set up in Lee's honour at the government-run Hong Kong Heritage Museum.