On Friday the military launched strikes to push the rebels out of Gwoza, the security sources said, and the garrison town of Damboa, which the militants sacked a month ago.
Both towns lie on roads linking Nigeria to Cameroon and Chad.
The military has struggled to stamp out the highly mobile, combat-hardened fighters of Boko Haram, which wants to carve an Islamic state out of religiously mixed Nigeria. The group is seen as the main security threat to Africa's biggest economy and leading energy producer.
"The special operation which began early in the week is meant to restore law and order to the area and apprehend all terrorists who have been operating in the locality," police spokesman Frank Mba announced on his website.
"The remnants of insurgents are being cleared from the communities. The mop up phase will ascertain the casualties."
Large sections of remote northeastern Borno state are under siege by the militants, who have killed more than 2,000 people this year - mostly civilians - and displaced hundreds of thousands more, rights groups say.
Witnesses and two security sources said heavily armed Boko Haram fighters stormed Gwoza, which lies in a rugged hilly area seen as a Boko Haram stronghold, on Wednesday.
Resident Abdullahi Abubakar said the insurgents sprayed the town with automatic gunfire, burning houses and overrunning the palace of its traditional ruler, the Emir of Gwoza. A security source said at least 30 bodies had been counted, with many more in the bush. He put the death toll at "several dozen."
"I could see the Boko Haram boys roaming the streets of Gwoza on motorbikes shouting 'Allahu Akbar' (God is Great), trying to shoot at us as we fled to the hill," Abubakar said by telephone. "Some of them were distributing food stuffs to the women and children left behind."
Senator Ali Ndume, who is from the area, said the new Emir of Gwoza, Idrissa Timta, who took over after the killing of his father in a Boko Haram attack in May, had been missing since the attack on his palace.
"The insurgents have sacked the town and people who are still there are locked indoors. Corpses litter the streets of Gwoza now but nobody can go out to pick them up," he said.
Abubakar said 500 refugees from the attack had spent three days in the hills around Gwoza surviving on wild fruit.
Footage obtained by rights group Amnesty International and released on Tuesday appeared to show Nigerian soldiers slitting the throats of Boko Haram suspects and dumping their bodies in a mass grave, mirroring how the Islamists treat their own captives and highlighting a runaway cycle of violence.
The five-year-old insurgency has been in the international spotlight since Boko Haram fighters kidnapped more than 200 girls from a school in the village of Chibok in April. The girls are still missing.