The warning means that the storm poses a threat to life and could inflict massive damage, the meteorological agency said.
The alert was issued for Mie prefecture, some 300 kilometres (190 miles) west of Tokyo, as the outer bands of the storm were already lashing the region and other areas of southwestern Japan, the agency said.
"Please remain on the alert against a rise of rivers, floods as well as damage from landslides in Mie prefecture," the agency said on its website.
Satoshi Ebihara, the Japanese weather agency's chief forecaster, also told a televised news conference that the levels of torrential rain monitored in Mie were already "unprecedented".
"We are in an abnormal situation where serious danger is imminent," Ebihara said.
"Please follow evacuation advisories from your local communities without any delay," he added. "Please do your best to protect your lives."
Yokkaichi and Suzuka, in northern Mie, issued evacuation instructions -- stricter than advisories but still not compulsory -- to some 512,000 residents in total, city officials and local media said.
"We are strongly urging our residents to evacuate, while we are hurriedly setting up temporary shelters across the city," a Yokkaichi official said.
Typhoon Halong, packing winds of up to 180 kilometres per hour, could make landfall in southwest Japan late Saturday or early Sunday, according to the agency.
Television footage showed high waves triggered by the typhoon splashing over breakwaters and muddy torrents roaring down a swollen river.
Storms and torrential rain earlier this week have left one dead and seven injured, public broadcaster NHK said.
At least 473 flights were cancelled due to the typhoon, which came as Japan had just begun its annual "Obon" summer holiday, NHK said.
Most ferry and train services on the southwest island of Shikoku were cancelled, while highways were closed at several points.
Over the next 24 hours, the storm was expected to dump 70 centimetres (28 inches) of rain on Shikoku, which had already been lashed by downpours from another typhoon last weekend, the national weather agency said.
The agency also warned of major landslides and floods in other areas of southwestern Japan, while local authorities in Tokushima in Shikoku issued an evacuation advisory to some 44,100 residents, officials said.
The typhoon, which was about 120 kilometres off Shikoku's southern tip at 0900 GMT, was moving northeast at 15 kilometres per hour, the agency said.
Halong comes a month after Typhoon Neoguri killed several people and left a trail of destruction in southern Japan.
Last weekend, a man drowned in a raging river while more than half a million people were advised to evacuate as heavy rain from Typhoon Nakri lashed the country.