John Martin ensured Leyton Orient's first victory at Gigg Lane for 33 years with an 87th minute winner.
And while it was hard on the home team Bury paid the penalty for missing a string of chances.
Orient, who have been a soft touch on their travels to the North West in recent years, created few opportunities.
However, when he got his moment Martin tucked away his shot superbly to put a dent in Bury's automatic promotion hopes.
However, the Shakers still found themselves moving up one place despite the defeat leaving manager Andy Preece to say: "It's our only consolation.
"We had enough chances to have won the game and if you don't take them then there's every chance you'll get punished.
"And they weren't difficult chances either and we did more than enough to have won the game." Matthew Joseph cleared a first-half effort from Jon Newby off the line while Newby and strike partner Colin Cramb wasted other chances.
In the second period Orient cut down the supply line and their worst moment came after 71 minutes when Michael Nelson forced a superb save from Glenn Morris.
A double substitution bringing on Chris Tate and Jabo Ibehre pepped up the visitor's attack.
And three minutes from time Martin struck to send the Orient fans home happy.
"The backbone of our victory was that we were solid in defence," said manager Paul Brush. "And we need to do that every week if we're to get more wins like this.
"The potential at this club is huge and it just shows what you can do with basic commitment and determination.
"I will back us against any side in this league so it's nice to know we can come to a place like Bury and get a result without playing at our best."