Jimmy Smith headed Russell Slade's men ahead in the 35th minute, but the Swans levelled on the stroke of half-time through Cedric Van der Gun.
Yet the real drama was still to come as Tate diverted a cross from substitute Paul-Jose M'Poku into his own net to send the League One club into the next round.
The visitors opened the scoring against the run of play 10 minutes before the break when Charlie Daniels swung in a free-kick from the right flank and Smith arrived to plant a firm header into the net from 12 yards.
Yet Swansea responded in style and a spirited rally led them to drawing level on the stroke of half-time when Van der Gun headed home from six yards out.
Orient rode a spell of second-half pressure and began to pose a threat of their own in the closing stages.
And with two minutes remaining, Tate inadvertently sent the ball past Yves Ma-Kalambay to send Orient into the fifth round.
Source: PA
Source: PA