According to reports, Mr Kittiratt was with a group of associates at the RBSC Polo Club, Bangkok.
The whistle-blowers were led by Thanat Thanakitamnuay, son of Kitti Thanakitamnuay, chief executive of property developer Noble Development.
"No matter what happens, I will not let this type of person have a place in the society," Mr Thanat said in front of the camera, before trying to get close to Mr Kittiratt.
As the whistle-blowers shouted and blew their whistles, Mr Kittiratt's associates intervened and both sides exchanged verbal abuse and a fight almost broke out.
During the commotion, Mr Thanat shouted, "Mr Kittirat, you must resign."
Another whistle-blower yelled, "You have time to hang out at the sports club but why don't you have time to help the farmers? Footballers don't hang themselves." He was referring to some cash-strapped rice growers who committed suicide because of the government's controversial rice pledging scheme.
Mr Kittiratt did not make any response to the whistle-blowers.