The National Police Agency credited its crackdown on the organised crime syndicates for membership falling to 58,600 last year, down from about 63,200 in 2012.
Like the Italian mob or Chinese triads, yakuza syndicates are involved in activities ranging from prostitution and drugs to extortion and white-collar crime.
Tougher policing, an increasingly poor public image and a slowing economy have made the lives of Japan's gangsters difficult, and turned membership less attractive for potential recruits, experts said.
Stronger laws in recent decades have also made it tough for mobsters to raise money for their operations, and even pushed some senior members to leave their organisations, Japanese media have reported.
However, some may have defected to organisations less known to police.
And the yakuza presence is still felt with the reputation of Japan's banking sector dented late last year after major lenders were entangled in an embarrassing loans-to-mobsters scandal.
According to Japan's police agency, the Yamaguchi-gumi, Japan's biggest crime group, lost 2,000 members from a year earlier with 25,700 gangsters on staff in 2013.
Rival group the Sumiyoshi-kai with 9,500 members, lost 1,100 members from a year before.