Copying my comment into an answer:
pdfTeX is (nearly always) the fastest engine, while LuaTeX (nearly always) has the most features. So if you care about speed and nothing else, then pdfTeX will always win, but if you want modern features and a well-maintained engine, then LuaTeX is usually the best choice. There's nothing wrong with XeTeX, but it is slower than pdfTeX and has fewer features than LuaTeX, so it usually only makes sense if the only feature that you need is OpenType fonts and you don't/can't use LuaTeX for some reason.
So if speed of compilation is your only concern, and you want to use New Computer Modern, then XeLaTeX is the best choice. But LuaTeX is recommended by the LaTeX Team for most new documents, and XeTeX is unlikely to be updated in the near future, and the very recent lua-unicode-math package significantly improves LuaLaTeX load times, so I'd generally recommend LuaLaTeX if you're willing to sacrifice a little bit of speed. I'm a little bit biased in favour of LuaTeX though, so while I think that LuaTeX is a better choice than XeTeX here, XeTeX is still a decent choice.
In your question, you mentioned that you also use pgfplots, and this is the only case that I'm aware of where LuaTeX is actually the fastest engine, because pgfplots can use Lua to accelerate its numerical computations. So if you draw enough complicated plots, LuaLaTeX may be the fastest option. But if you only draw simple plots, then the general slowness of LuaLaTeX will probably offset any gains from pgfplots, so this probably won't make much of a difference for typical documents.