Well, Handbrake and Mpeg Streamclip both have pretty good reputations in speed of converting. Although I find the quality vs file size comparison often in Mpeg Streamclip's dis-advantage.
Just wondering:
Did you export an 'edited' timeline, with all your effects etc. on it in FCP X, or did you just dropped the finished movie in an empty timeline?
If you did the first, could you try to do the second? Dropping the finished movie and rendering to the Apple TV setting in FCP X? See if that makes it render faster...
And is the quality of both comparable?
Anyway, this doesn't completely surprise me. In some areas I find FCP X very fast (scrubbing footage, working with H.264, ...) but in others I find it very slow (working/exporting with XDCAM, exporting some cut cut- edits, ...)