Unicode fonts are large. Loading them takes a significant amount of time.

tikz and pgfplots are slow, especially if you use pgfmath a lot.

If you have many or complex pictures/plots, you can greatly speed compilation by externalising them. The standard method uses the external library, but this is neither maintained nor robust. A better solution these days is either memoize or robust-externalize. I have only used memoize, but either package should help in the situation you describe.

memoize currently requires the installation of additional software for maximum benefit. (This should change soon, but not yet.) It does not need shell-escape and can deal with many situations out-of-the-box. For example, it does not need any setup to memoize tikzpictures or to handle cross-references, citations etc. correctly. If something doesn't work out-of-the-box, configuration is usually straightforward but often non-obvious. That is, mostly, it is just a line of key-value form, but it is often not at all clear which key or which value are the ones you want.

robust-externalize I believe requires shell escape, but I don't think it requires additional software. I don't know what the configuration is like.

Note that I am biased towards memoize, so you should investigate both options for yourself. The one thing I feel no compunction in urging is to avoid using the standard external library. Its own developers recommend using memoize or robust-externalize now, since these are both much more recent, actively developed and comparatively robust solutions.