It all looked so bright for the visitors when Graham Turner's side took the lead on 13 minutes, defender Dean Beckwith headed home Simon Johnson's corner, but then the O's hit back to level through John Melligan and then wrap up the points with Boyd's spot-kick.
On a miserable, rain-soaked afternoon, it was Orient, one of the bookies' favourites to go down, who started well.
Skipper Stephen Purches went close early on as Melligan caused all sorts of problems for the visitors.
Melligan, who has been taken off the transfer list following some impressive pre-season performances, was a menace throughout and after a good passage of play from the home side Jason Demetriou and Paul Terry went close to opening the home side's account.
So, it came as a bolt out of the blue when Hereford took the lead on 13 minutes.
Johnson's corner from the left was headed goalwards by defender Beckwith and the ball took a wicked deflection past Glenn Morris in the home goal.
Orient levelled 11 minutes later when Melligan floated in a harmless cross from the right, Boyd almost got a touch as he challenged Hereford keeper Darren Randolph and the ball tamely drifted into the net.
Demetriou should have put Orient ahead, but fired wildly over before the home side were rewarded on 45 minutes when Richard Rose handled in the box and Boyd placed the spot-kick into the right-hand corner.
The second half was an instantly forgettable affair as Adam Chambers and Ryan Jarvis had efforts for Orient.
Although Hereford had plenty of possession, they were restricted to long-range strikes from Kris Taylor and Matt Done that were comfortably saved by Morris.