No one was injured in the fire, but the smoke was enough to bring three fire companies to the scene.
Fire fighters believe the fire started when someone outside the restaurant flicked a cigarette butt near the wall of the front of the building.
The cigarette then lit the inside of the wall and smoke poured into the building.
Employees evacuated the crowded restaurant.
"We started smelling something that was burning," said Jennifer Sawtelle, an Olive Garden employee. "The smoke alarm went off and we evacuated. The whole restaurant was filled because of Father's Day," Sawtelle added.
"They were all evacuated," said Kearsarge Fire Chief Bill Nichols. "They did a nice job evacuating the building cause there was a lot of smoke in there when I got here," Nichols added.
Crews put out the fire in less than an hour.
Fire officials estimated the damage at about $5,000.
There is no word on when the Olive Garden will reopen.