Copying my comment into an answer:
pdfTeX is (nearly always) the fastest engine, while LuaTeX (nearly always) has the most features. So if you care about speed and nothing else, then pdfTeX will always win, but if you want modern features and a well-maintained engine, then LuaTeX is usually the best choice. There's nothing wrong with XeTeX, but it is slower than pdfTeX and has fewer features than LuaTeX, so it usually only makes sense if the only feature that you need is OpenType fonts and you don't/can't use LuaTeX for some reason.
So if speed of compilation is your only concern, and you want to use New Computer Modern, then XeLaTeX is the best choice. But LuaTeX is recommended by the LaTeX Team for most new documents, and XeTeX is unlikely to be updated in the near future, and the very recent lua-unicode-math package significantly improves LuaLaTeX load times, so I'd generally recommend LuaLaTeX if you're willing to sacrifice a little bit of speed. I'm a little bit biased in favour of LuaTeX though, so while I think that LuaTeX is a better choice than XeTeX here, XeTeX is still a decent choice.