"I deplore the loss of life in (the village of) Du Chee Yar Tan and call on the authorities to carry out a full, prompt and impartial investigation and ensure that victims and their families receive justice," Ms Pillay said in a statement.
The UN, the statement said, had received "credible information" that eight Rohingya Muslim men were attacked and killed in the village, near the border with Bangladesh, by local Rakhine Buddhists on Jan 9.
Four days later, a police sergeant in the same village was captured and killed by Rohingya.
This in turn prompted village police and local Rakhine to kill at least 40 Rohingya Muslim men, women and children the same evening, the statement said, adding that the UN had passed on the information it had received to the Myanmar government.
"By responding to these incidents quickly and decisively, the government has an opportunity to show transparency and accountability, which will strengthen democracy and the rule of law in Myanmar," Ms Pillay said.
Myanmar's western Rakhine state remains tense after several outbreaks of communal bloodshed between Buddhist and Muslim communities since 2012 that have killed scores and displaced 140,000 people, mainly from the Rohingya minority.
Details of the latest unrest have remained unclear, but activists said shortly after the Jan 13 attack that at least two women and a child were stabbed to death in the village, with possibly several dozen casualties.
Myanmar authorities have denied any civilian deaths but confirmed a clash took place in which a police officer was presumed to have been killed.