The companies, which have a combined market value of about US$28 billion, entered into a 90-day exclusive agreement to negotiate and finalise the price and structure of the merger, according to a joint statement released on Friday.
The plan includes the creation of a major Islamic bank, they said.
A combination of the three banks would create a group with total assets of 597.3 billion ringgit ($188 billion), surpassing the assets worth 560.4 billion ringgit of Malayan Banking Bhd (Maybank).
The latest move comes three years after Kuala Lumpur-based CIMB, which also has operations in Thailand, scrapped talks to purchase RHB Capital.
"We believe there is inherent value and that there will be many opportunities which will arise from this proposed merger," Kellee Kam, group managing director of RHB Capital, said in the statement. "We hope to explore this further through our discussions."
Employees Provident Fund, Malaysia's biggest pension manager, owns a 14.5% stake in CIMB and a 41.3% interest in RHB Capital. The fund owns 65% of Malaysia Building Society.
"A bigger-sized bank with a more solid footing in the domestic market will be in a better position to grow," said Harsh Wardhan Modi, an analyst at JPMorgan Chase in Singapore. "Malaysia has always been an intensely competitive market."
CIMB and Maybank, Malaysia's two biggest lenders, scrapped separate talks to acquire RHB Capital in June 2011 over concerns about price, people familiar with the matter said at the time.