The Congolese striker thumped the spot-kick home after Matt Jarvis had been felled by Justin Miller in the area.
Orient had led through James Walker's spectacular first-half strike, but after Luke Guttridge was dismissed in the 71st minute, Brian Saah put through his own net before Ndumbu-Nsungu snatched the winner.
Ronnie Jepson's side had begun the stronger but Jarvis, Gary Mulligan and Bas Savage all failed to fully test Orient keeper Glyn Garner with their efforts.
Orient struggled to retain possession and were eventually carved open in the 30th minute. Savage showed great strength to turn his marker and find Andrew Crofts waiting for the cross, only for the Welsh international to bring a superb reaction save out of Garner.
But just five minutes later, after Garner had denied Jarvis with another fine save with his feet, the O's burst down the field and stunned Gillingham with a classic counter-attacking move to take the lead. Guttridge supplied a fine pass to Walker and the striker curled a precise finish into the far corner from 20 yards.
Moments later Guttridge again had Gillingham panicking when he let fly from long-range but saw his effort fly over.
After Gills' captain Michael Flynn blazed an effort over himself, Matt Lockwood's 45th minute corner forced a goalmouth scramble that two Orient players failed to convert on the line.
Orient came out firing after half-time and, on 56 minutes, had a goal disallowed when John Mackie was adjudged to have fouled keeper Dean Brill when heading home. And moments later Brill had to be alert to tip over Adam Chambers' dipping long-range volley.
But the game turned on 71 minutes when Guttridge saw red for a second bookable offence and Orient soon came under intense Gillingham pressure.
Crofts was denied by another point-blank Garner save but in the 81st minute Ian Cox's searching cross deflected off defender Saah and into the corner.
The Gills went in search of the winner and were rewarded when Jarvis was brought down by Miller in the area and Ndumbu-Nsungu stepped up to send Garner the wrong way and deny Orient a precious point.