Protesters torched a number of diplomatic vehicles as security forces used water cannons and warning shots in a bid to drive them out.
Police had earlier fired warning shots to disperse a few thousand of protesters as they approached the main gate of the mission.
"O, Allah's messenger... O, Mohammed," protesters chanted.
The attack comes two days after four Americans including the ambassador were killed when a Libyan mob attacked the US consulate in Benghazi, and protesters in Cairo tore down the Stars and Stripes and replaced it with a black Islamic flag.
The low-budget movie, "Innocence of Muslims" in which actors have strong American accents, portrays Muslims as immoral and gratuitously violent.
It pokes fun at the Prophet Mohammed and touches on themes of paedophilia and homosexuality, while showing him sleeping with women, talking about killing children and referring to a donkey as "the first Muslim animal".
Meanwhile, Washington has deployed warships and Marines to Libya in the wake of the slaying of its ambassador. A detachment of 50 Marines was dispatched to secure the main American embassy in the Libyan capital Tripoli, where staff numbers were being cut to emergency levels, United States officials said.
The US also began evacuating all its staff from its mission in Benghazi, where 52-year-old ambassador Chris Stevens, a fervent supporter of the revolt that overthrew Moamer Kadhafi, and three other Americans were killed on Tuesday.
At the same time two destroyers were sent to "the vicinity of Libya" as a precautionary measure, a senior US official said.
"Make no mistake, we will work with the Libyan government to bring to justice the killers who attacked our people," US President Barak Obama said, while ordering increased security at US diplomatic missions around the world.
Tuesday's assault on the US consulate in Benghazi followed a violent protest at the US embassy in Cairo over an amateurish anti-Islamic film made in the United States and reportedly promoted by a group of US-based Egyptian Copts.
Initial reports said Stevens and the three other Americans were killed by an angry mob as they tried to flee in a car.
But it is now believed that Stevens died from smoke inhalation after becoming trapped in the compound when suspected Islamic militants fired on the building with rocket-propelled grenades and set it ablaze.
US officials are investigating the possibility that the assault was a plot by Al-Qaeda affiliates or sympathisers, using the protests as a diversion to carry out a coordinated revenge attack on the 11th anniversary of 9/11.
US Secretary of State Hillary Clinton said the killings should "shock the consciences" of people of all faiths but vowed that the actions of "a savage and small group" would not make Washington turn its back on Libya.
Obama ordered security tightened at all US diplomatic missions, amid fears that anti-American protests sparked by the low-budget, privately produced film denigrating the Prophet Mohammed could spread.