The deal was announced in a press statement as Japanese Prime Minister Shinzo Abe and visiting Australian Prime Minister Tony Abbott held a briefing in Tokyo Monday evening.
It "will create significant new trade and investment opportunities for the two countries", the statement said, adding that efforts would be made to sign a final deal "as early as possible".
The announcement comes with Canberra also set to sign a free trade pact with Seoul on Tuesday after four years of negotiations. Abbott will head to China after South Korea as part of an East Asian tour.
Under Abe, Tokyo has entered into talks on the massive Trans-Pacific Partnership, a proposed free-trade deal that would encompass 12 nations including the United States and Japan.
But there are major sticking points among various nations, including the opening of protected domestic markets such as agriculture and automobiles.
Japan has long been accused of protecting its domestic industries -- including the politically powerful agricultural sector -- with high trade barriers, while many of its own exports, including vehicles and electronics, enjoy big sales overseas.
The US has expressed frustration with Japan over its stance on keeping certain sectors out of the Trans-Pacific Partnership, as talks continue.
Tokyo is also in separate free-trade negotiations with the European Union.
Australia would become the first major exporter of farm produce, including beef, to conclude a free-trade accord with Japan, Kyodo news agency said, with a deal expected to give Australian exports a significant competitive edge over US rivals.
- 'The Asian century' -
There were few details announced in the statement released Monday.
But Japanese media have reported that Australia would drop its five percent duty on small and mid-sized Japanese cars while Tokyo is ready to lower its tariff on Australian beef, currently sitting at 38.5 percent.
Abe and his Australian counterpart also talked about cooperation in security, including joint development of defence equipment, after Japan last week loosened a self-imposed ban on weapons exports to boost Tokyo's global role in a move which unnerved neighbouring China.
The Australian premier attended a national security council meeting in Tokyo on Monday afternoon, a first for a foreign leader.
"This will be the Asian century," Abbott said earlier Monday.
"I think that's true: The better Asia is, the better the world will be," he added.
Regional tensions have soared as China and Japan lock horns over the ownership of a string of islands in the East China Sea, while Beijing is also embroiled in a dispute with several nations over territory in the South China Sea, which it claims almost in its entirety.
An unpredictable North Korea looms over the regional power balance.
Abbott's visit came as Tokyo said last week it would cancel its annual Antarctic whaling hunt for the first time in more than 25 years to abide by a UN court ruling that the scheme was a commercial activity disguised as science.
Australia, backed by New Zealand, hauled Japan before the International Court of Justice in 2010 in a bid to end the annual Southern Ocean hunt -- a thorny diplomatic spat that threatened to damage the trade talks.
The deal with South Korea, meanwhile, calls for Canberra and Seoul to remove almost all tariffs on traded goods within 10 years of the agreement going into effect.
Australia will abolish a five-percent import tariff on most South Korean-made cars while a five percent tariff on other South Korean exports such as TVs, refrigerators and machinery would also be eliminated immediately.
Tariffs will also go on resources, energy and manufactured goods, while the deal will open the door to new opportunities for Australian firms in South Korea's education and telecommunications markets.