Gylfi Sigurdsson, back at Swansea after signing from Tottenham Hotspur, fired the winner from close range after 72 minutes to stun the hosts, who had drawn level through skipper Wayne Rooney shortly after halftime.
Hopes were high for a winning start for the 20-time English champions after last season's miserable seventh-place showing, United's worst since 1990, but the problems they suffered under David Moyes were evident again.
United had plenty of possession in the first half but lacked invention and pace and Swansea took the lead with their first real attack when Korean midfielder Ki Sung-yueng slotted home after 28 minutes.
The goal inspired the travelling Swansea supporters to start chanting "You're getting sacked in the morning!" at Van Gaal, who sat impassively in his seat in the dugout.
Rooney levelled with an overhead flick after 53 minutes and struck the post with a curling free kick but Swansea took advantage of some poor defending by United left back Ashley Young to pocket three unlikely points.
United had to make do without Robin van Persie, Michael Carrick and Danny Welbeck among others for the match, a scrappy affair that featured six yellow cards, four of them to the visitors.
Injuries obliged Van Gaal to give competitive debuts to Tyler Blackett, 20, and 21-year-old Jesse Lingard.Lingard's day was to last only 24 minutes, however, as he was forced off after appearing to jar his knee in a challenge with visiting captain Ashley Williams.
Van Gaal introduced Nani in place of striker Javier Hernandez at halftime, and United responded immediately. Juan Mata's right-wing corner was flicked on by Phil Jones and Rooney hooked home a spectacular close-range volley at the back post.
But it was Swansea who claimed the winning goal when Icelandic midfielder Sigurdsson slammed home from Wayne Routledge's mishit volley.
The win was only the second at Old Trafford in the history of Swansea, who claimed their first just last season in an FA Cup third-round tie.