ANA Holdings - the airline's parent company - announced last year that it planned to invest about $25 million in Asian Wings Airways.
But "competition between new and old airlines in Myanmar has intensified... calling into question the assumptions made at the time of the original decision," the Japanese firm said in a statement Wednesday. "Ultimately, negotiations for the capital participation with AWA were unable to reach an agreement, and the investment plan was cancelled as a result."
The move will not impact earnings for the current fiscal year, ANA said.
The announcement came as the Japanese carrier said it had swung back to profitability in the three months to June. With little room for growth in the domestic market, ANA and other Japanese firms have been eyeing Southeast Asia as a lucrative market.
Foreign companies have piled into Myanmar since the installation of a nominally civilian government in 2011, eager to make the most of opportunities in the country as it opens up following decades of junta-led government.
Shares in ANA were 3.4% higher at 259.2 yen on the Tokyo Stock Exchange Thursday morning.