On four minutes Rushden went close when Alun Armstrong saw his shot deflected and Glyn Garner was forced to make an excellent save.
Four minutes later David Bell wasted a good chance for Rushden when he shot well over when well placed.
Leyton retaliated in the 12th minute when Daryl McMahon saw his snap-shot go inches wide and three minutes later Jamie Young in the Rushden goal just held on to Matt Lockwood's drive from outside the area.
On 27 minutes Rushden almost went ahead when Drewe Broughton drew a smart save by Garner.
On 34 minutes Bell crossed from the right and Armstrong's header was superbly tipped round the post by Garner, while just before the interval Lockwood almost got on the end of a superb cross by Shane Tudor.
On 48 minutes Broughton's flick header just missed by Savage who had stole in at the far post.
Play became scrappy with both sides trying to gain an initiative but on 61 minutes Marcus Kelly was inches wide after a good Rushden build up.
One minute later Orient had an appeal for a penalty turned down after Andy Burgess appeared to push Gary Alexander.
Orient grabbed the winner in the 66th minute when Donny Barnard crossed for Alexander whose shot was well saved by Young but the ball fell to the feet of substitute Lee Steele who scored from close range.
Rushden fought hard for the equaliser and on 71 minutes Kelly was inches wide following a cross from Bell.
Orient, however, had taken confidence from their goal and looked the better side in the closing stages.
Young was forced to produce a brilliant save to deny Alexander in the 77th minute.
On 87 minutes Alexander almost put the game beyond doubt when he was inches wide with a fierce shot.