In 1974, as US troops withdrew from Vietnam, China invaded the Paracel Islands, which were held by the US-backed South Vietnamese regime.
More than 70 Vietnamese soldiers died during the invasion and China has controlled the island chain ever since.
The two countries are locked in long-standing territorial disputes over the Paracel and Spratly islands, which both claim, and often trade diplomatic barbs over oil exploration and fishing rights in the contested waters.
On Sunday, dozens of activists laid flowers at a statue of Ly Thai To -- the founder of Hanoi and a nationalist figurehead -- in the centre of the capital.
Activists waved banners and shouted "Hoang Sa (Paracels), Truong Sa (Spratlys) belong to Vietnam!" before hundreds of uniformed and plain clothed police forced them to leave the area.
"We gathered here to commemorate the event... Forty years ago the Chinese invaded the island and killed many Vietnamese soldiers," academic Nguyen Quang A told AFP at the event.
The protest was the first display of public discontent in Hanoi this year against Beijing's perceived aggression over territory, following a handful of anti-China demonstrations last year which were broken up by authorities.
"The government of Vietnam is in a very difficult situation," Quang A said, calling the heavy police presence at the event "ridiculous".
"The memory of people of Vietnam is vivid. Nobody can eradicate that memory," he said.
Vietnam's tightly-controlled state media covered the anniversary but not Sunday's protest. There was no official comment from the government.
Beijing's increasingly assertive stance in the South China Sea has stoked public anger in Vietnam and given way to rare protest in the authoritarian country.
Apart from China and Vietnam the Spratly Islands are claimed in whole or in part by the Philippines, Malaysia, Brunei and Taiwan.