The farmers travelled by buses from in front of the Commerce Ministry, where they have been encamped, and arrived at Muang Thong Thani about 11.30am.
Police and military personnel were deployed around the defence building and razor wire barricades erected to keep the protesters out.
They farmers demanded to meet the prime minister, who reportedly arrived at the office about 10.30am.
The farmers were reported to have been able to break through the wire barrier and to have entered the compound, but they stayed outside the building.
Rawee Rungruang, chairman of the network of farmers from western provinces, insisted that he would talk only to the prime minister.