The militants launched an attack on the town of Bama, 70 km (45 miles) from the Borno state capital of Maiduguri, on Monday.
They were initially repelled but came back in greater numbers overnight, the sources and witnesses said.
Nigerian defence spokesman did not respond to a request for comment. The sources said there were heavy casualties on both sides and one said at least 5,000 people fled the town.