The Serb came from a set down to win 4-6, 6-3, 6-3, ending Williams' bid to match Chris Evert and Martina Navratilova with 18 Grand Slam titles.
Williams, who was gunning for a sixth Australian crown, did not appear to be moving freely with unconfirmed reports that she was carrying an injury.
She pulled out of the doubles on Friday, although that was blamed on an injury to sister Venus.
It was a major scalp for Ivanovic, 26, and set up a last-eight clash against either Canadian teenager Eugenie Bouchard or local hope Casey Dellacqua.
"It's amazing, I just wanted to play the best I could," said Ivanovic. "I'm so thrilled that I played so well. I was very consistent and I'm happy about that."
Since winning at Roland Garros in 2008, Ivanovic, ranked 14, has only made a Grand Slam quarter-final once before, at the 2012 US Open.
But under new coach Nemanja Kontic, the Serb has been thriving and came into Melbourne on the back of winning her 12th career title in Auckland.
She ground down Williams with her sizzling groundstrokes for her first win against the American in five encounters.
Despite the shock loss, Williams set a new record by appearing in her 70th match at the Australian Open, surpassing the previous record held by Lindsay Davenport.
Williams, 32, started with two double faults, helping Ivanovic to three break point chances, but the world number one saved them all and regrouped to win the game.
The serve of Ivanovic lacked the power of her opponent but she mixed it up enough to hold.
Williams blasted through her next service game and Ivanovic who has family in Melbourne and is popular with Australian fans, earning her the nickname "Aussie Ana", rallied from two points down in the next to hold.
The Serb was moving well and a cross-court forehand gave her a break point on the next Williams serve and she grabbed the chance, with the American sending a backhand into the net.
But her advantage didn't last long with a poor service game allowing Williams to level.
The cool Williams held serve to be 4-3 in front, yelling her first "C'mon!" as she pumped herself up.
Ivanovic took it to 4-4 and Williams' lightning fast serves saw her hold before she turned up the heat and took the set on her second break point.
The second set went to serve until the fifth game when Ivanovic worked her way to two break points. Williams saved one with a towering serve but she lost the other, failing to bury a forehand into an open court.
The Serb had another two break points in the seventh game as Williams came under increasing pressure, but she rose to the occasion, once again saving.
For the third Williams serve in a row, Ivanovic worked her way to break points and while the American again saved two, she couldn't stop a raking forehand winner on the third.
Williams was being worn down and the Serb scored a big break to go 2-0 in front in the third set and then held serve to take control.
The despondent American was broken again to go 4-1 and while she saved one match point in the eighth, Ivanovic served to love for a famous win.