They said 50 passengers had been rescued after the boat sank around 30 kilometres (20 miles) south of the capital Dhaka.
"We have heard that the ferry was overloaded with passengers and the river was rough," local police chief Tofazzal Hossain told AFP.
The police chief of Madaripur, where the ferry began its journey, said it was carrying between 170 and 200 passengers.
Ferry sinkings are common in Bangladesh, with overcrowding and poor ship design and maintenance often to blame.
Bangladesh, one of Asia's poorest nations, is criss-crossed by more than 230 rivers and boats are the main form of travel, especially in the southern and northeastern regions.