The difference you observe between XeTeX and LuaTeX is explained by the following excerpt from mathastext's doc
edit this excerpt being taken out of context, the first sentence is a bit ambiguous; it is to be read as "XeTeX may create italic corrections but it does not follow Appendix G of the TeXBook in their use in math mode, contrarily to LuaLaTeX". I have examined this further now (four years later) and report below on some interesting findings also regarding pure text mode.
From brief testing on 2012/12/28, XeTeX seems not to create fake
italic corrections for OpenType fonts. Hence the TeX placement
algorithms for math mode described in this section do not work well
when an OpenType (text) font is used for the letters in math mode, and
the document is compiled with the XeTeX engine. On the other hand
LuaLaTeX seems to implement the italic corrections when using OpenType
fonts, but only with italic fonts (as far as I could tell). Try the
following (which will use the OpenType Latin Modern font) on a recent
TeX installation and compare the output of both engines:
\documentclass{article}
\usepackage{fontspec}
\begin{document}
\Huge
$\mathit{f_i^i}$\par $\mathrm{f_i^i}$
\end{document}
Comment out the fontspec line and use pdfLaTeX. All three outputs are
different on my TeX installation. XeTeX does not have the italic
corrections. LuaLaTeX does, but only for the italic font. pdfLaTeX has
them for both the italic and the upright font.
I checked again about one year ago and nothing had changed. On your example I checked that the better spacing relative to the right delimiters is indeed explained by the insertion by LuaLaTeX of italic corrections.
Here is some additional experiment I did now about whether italic corrections are defined and whether they are used or not in math mode.
According to the TeXBook if the \fontdimen2 parameter vanishes, then italic corrections should be used when letters follow one another. This is done by LuaLaTeX but not by XeTeX. Furthermore there is another problem that the italic corrections defined by XeTeX seem to differ from the ones defined by LuaLaTeX (this shows also as a pure text mode difference, cf the different widths of hboxes below 34.18512pt vs 35.18033pt for \hbox{f\/g\/k)}).
Test document:
% Time-stamp: <03-11-2016 15:07:54 CET>
\documentclass{article}
\usepackage{fontspec}
\usepackage{ifluatex}
\begin{document}
\Huge
% $\mathit{f_i^i} \mathit{(f\/)}$\par $\mathrm{f_i^i}$
\ifluatex
\usefont {EU2}{lmr}{m}{it}
\else
\usefont {EU1}{lmr}{m}{it}
\fi
\leavevmode\hbox{fgk}
\leavevmode\hbox{f\/g\/k}
\typeout{\the\fontdimen2\font}% 8.70801pt in both engines
$\mathit{fgk}$
\fontdimen2\font 0pt
$\mathit{fgk}$
\showoutput
\end{document}
Log with XeTeX:
....\hbox(17.51552+5.0755)x29.856
.....\EU1/lmr/m/it/24.88 fgk
....\penalty 10000
....\glue(\parfillskip) 0.0 plus 1.0fil
....\glue(\rightskip) 0.0
...\glue(\parskip) 0.0 plus 1.0
...\glue(\baselineskip) 7.40898
...\hbox(17.51552+5.0755)x345.0, glue set 295.81488fil
....\hbox(0.0+0.0)x15.0
....\hbox(17.51552+5.0755)x34.18512
.....\EU1/lmr/m/it/24.88 f
.....\kern 3.63248
.....\EU1/lmr/m/it/24.88 g
.....\kern 0.69664
.....\EU1/lmr/m/it/24.88 k
....\penalty 10000
....\glue(\parfillskip) 0.0 plus 1.0fil
....\glue(\rightskip) 0.0
...\glue(\parskip) 0.0 plus 1.0
...\glue(\baselineskip) 7.40898
...\hbox(17.51552+5.07552)x345.0, glue set 300.144fil
....\hbox(0.0+0.0)x15.0
....\mathon
....\hbox(17.51552+5.07552)x29.856
.....\EU1/lmr/m/it/24.88 glyph#55
.....\EU1/lmr/m/it/24.88 glyph#59
.....\EU1/lmr/m/it/24.88 glyph#70
....\mathoff
....\penalty 10000
....\glue(\parfillskip) 0.0 plus 1.0fil
....\glue(\rightskip) 0.0
...\glue(\parskip) 0.0 plus 1.0
...\glue(\baselineskip) 7.40897
...\hbox(17.51552+5.07552)x345.0, glue set 300.144fil
....\hbox(0.0+0.0)x15.0
....\mathon
....\hbox(17.51552+5.07552)x29.856
.....\EU1/lmr/m/it/24.88 glyph#55
.....\EU1/lmr/m/it/24.88 glyph#59
.....\EU1/lmr/m/it/24.88 glyph#70
....\mathoff
Log with LuaLaTeX:
nota bene: the \kern1.59232 before the \mathoff (after having set to zero the \fontdimen2 parameter) is the italic correction for the letter k. XeTeX puts no such italic correction and furthermore the value it assigns as italic correction for the k is 1.09473pt (in 24.88 point size) not the 1.59232pt of LuaLaTeX.
(try with \leavevmode\hbox{k\/}
> 12.29073-1.09473;
11.19600
> 12.78831-1.59232;
11.19599
)
....\hbox(0.0+0.0)x15.0, direction TLT
....\hbox(17.51552+5.07552)x29.856, direction TLT
.....\EU2/lmr/m/it/24.88 f
.....\EU2/lmr/m/it/24.88 g
.....\EU2/lmr/m/it/24.88 k
....\penalty 10000
....\glue(\parfillskip) 0.0 plus 1.0fil
....\glue(\rightskip) 0.0
...\glue(\parskip) 0.0 plus 1.0
...\glue(\baselineskip) 7.40897
...\hbox(17.51552+5.07552)x345.0, glue set 294.81967fil, direction TLT
....\localpar
.....\localinterlinepenalty=0
.....\localbrokenpenalty=0
.....\localleftbox=null
.....\localrightbox=null
....\hbox(0.0+0.0)x15.0, direction TLT
....\hbox(17.51552+5.07552)x35.18033, direction TLT
.....\EU2/lmr/m/it/24.88 f
.....\kern 4.13008
.....\EU2/lmr/m/it/24.88 g
.....\kern 1.19424
.....\EU2/lmr/m/it/24.88 k
....\penalty 10000
....\glue(\parfillskip) 0.0 plus 1.0fil
....\glue(\rightskip) 0.0
...\glue(\parskip) 0.0 plus 1.0
...\glue(\baselineskip) 7.40897
...\hbox(17.51552+5.07552)x345.0, glue set 298.5517fil, direction TLT
....\localpar
.....\localinterlinepenalty=0
.....\localbrokenpenalty=0
.....\localleftbox=null
.....\localrightbox=null
....\hbox(0.0+0.0)x15.0, direction TLT
....\mathon
....\hbox(17.51552+5.07552)x31.44832, direction TLT
.....\EU2/lmr/m/it/24.88 f
.....\EU2/lmr/m/it/24.88 g
.....\EU2/lmr/m/it/24.88 k
.....\kern1.59232
....\mathoff
....\penalty 10000
....\glue(\parfillskip) 0.0 plus 1.0fil
....\glue(\rightskip) 0.0
...\glue(\parskip) 0.0 plus 1.0
...\glue(\baselineskip) 7.40897
...\hbox(17.51552+5.07552)x345.0, glue set 293.22736fil, direction TLT
....\localpar
.....\localinterlinepenalty=0
.....\localbrokenpenalty=0
.....\localleftbox=null
.....\localrightbox=null
....\hbox(0.0+0.0)x15.0, direction TLT
....\mathon
....\hbox(17.51552+5.07552)x36.77264, direction TLT
.....\EU2/lmr/m/it/24.88 f
.....\kern4.13008
.....\EU2/lmr/m/it/24.88 g
.....\kern1.19424
.....\EU2/lmr/m/it/24.88 k
.....\kern1.59232
....\mathoff
The initial answer was edited to remove the action on operator names such as cos, sin, etc...
n.b.: mathastext (whose use is not recommended for OpenType fonts, but I am trying to use its rather sophisticated mechanism of math active letters here) has no official interface for different fonts for "upright" and "italic", the "italic" one (if used) being supposed to differ from the "upright" one only via the NFSS shape specifier (usually "it" or "sl".) Here we use directly EB Garamond 12 Italic, but for mathastext it receives it from fontspec as an "upright" font, ie the NFSS shape specifier is "n". This is why operator names were also using the Italic font. Hence the edit reverts the corresponding action of mathastext on the font used for operator names.
A further side remark is that mathastext reserves two mathgroups. Here only one is actually needed, but there is no option to tell it to mathastext currently. In fact we could even make do without any additional mathgroup as the needed ones are already declared by unicode-math. But again, mathastext is primarily for pdftex use with traditional TeX fonts.
\documentclass{standalone}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{fontspec}
\usepackage[math-style=ISO,bold-style=ISO,nabla=upright,partial=upright]{unicode-math}
\def\basemathfont{Latin Modern Math}
\def\myfont{EB Garamond}
\def\mymathfont{\myfont}
\def\myitalicfont{ 12 Italic}
\setmainfont[Ligatures=TeX,
RawFeature={+calt,+onum},
ItalicFeatures={RawFeature={+calt,+dlig,+onum,+cswh,+swsh}}]{\myfont}
\let\amp\&
\renewcommand{\&}{{\fontspec{EB Garamond}\textit{\amp}}}
\setmathfont{\basemathfont}
\setmathfont[range={\mathup/{latin,Latin,greek,Greek,num},\partial,\mathpunct,\mathbin,\mathrel,\infty,\mathpercent,\mathampersand,\mathdollar,\mathquestion},CharacterVariant={5,11},Numbers=Lining]{\mymathfont}
\setmathfont[range={\mathit/{latin,Latin,greek,Greek,num},\ell},CharacterVariant={5,11},Numbers=Lining]{\mymathfont \myitalicfont}
% load mathastext minimally
\makeatletter\let\operator@fontORI\operator@font\makeatother
\usepackage[basic, nodigits, nohbar, defaultimath,
defaultalphabets, defaultmathsizes]{mathastext}
\makeatletter\let\operator@font\operator@fontORI\makeatother
% next line after looking in log file, there is surely better way, but I don't
% know if unicode-math has public interface for fetching the assigned
% NFSS font family name
\MTfamily{EBGaramond12Italic(0)}\Mathastext
% Perhaps an alternative would be to to do
% skip the initial \setmainfont and issue instead
% \setmainfont[CharacterVariant={5,11},Numbers=Lining]{\mymathfont
% \myitalicfont}
% then:
% \usepackage[...]{mathastext}
% then:
% the correct \setmainfont.
% but I don't know how unicode-math interacts with the setmainfont setting
%
% mathastext does not know that the font is "italic" (from the NFSS point of
% view it is defined with normal not italic shape) and it will thus try
% to insert "italic corrections" (cf documentation.) Get rid of that:
\MTnoicinmath
% Now we can use mathastext's facilities:
\MTsetmathskips{f}{5mu}{5mu}
\MTsetmathskips{j}{5mu}{3mu}
% Nota Bene: such skips are ignored inside inside \math... commands
\setmathfont[range={\otimes,\oplus,\square,\boxtimes,\boxminus,\boxplus,\cdot,\nabla,\ll,\gg,\lll,\ggg,\mathcolon,\Rightarrow,\Longrightarrow,\Leftrightarrow,\Longleftrightarrow}]{\basemathfont}
\setmathfont[range={\mathcal,\mathscr,\mathbfscr,
"2AA1,"2AA2,"2AF7,"2AF8,\lesssim,\gtrsim,\in,\notin,
\sumint,\equiv,\mp,\prime,\dprime,\trprime,\propto,\sim,\rightarrow,\leftarrow,\leftrightarrow,
\dots,\ddots,\vdots,\parallel,\perp}]{XITS Math}
\AtBeginDocument{\Umathcode`\* = 0 0 "002A\relax}
\DeclareSymbolFont{AMSb}{U}{msb}{m}{n}
\protected\def\mathbb#1{{\mathchar\numexpr256*\symAMSb+`#1\relax}}
\begin{document}
$(j)$
$(f)$
$[f]$
$\{f\}$
$\displaystyle\left.\frac{\mathrm{d}}{\mathrm{d}x}f\right|_{x=4}$
\end{document}
% Local Variables:
% TeX-engine: xetex
% End:
Produces on my TL2016:
You can adjust to your liking the extra skips on left and right of the letters via \MTsetmathskips as illustrated above only for f and j.
I tried with luatex (using luatex85), this is the result with your original code (everything mathastext related commented out), on my TL2016:
Notice how the two derivatives differ, the rendering in XeTeX is weird (see the "d" on top; this has nothing to do with use of mathastext, it gives the same without). And the fraction sign width seems to differ.
Another thing is that for some reason my equal sign is not like in your second picture. The one in my picture uses the EB Garamond from TeXLive.
A question about $ij$ was raised in a comment. I had forgotten about it but you can use the \Mathnormal defined by mathastext: within it the extra skips configured via its command \MTsetmathskips are cancelled. Here is with the \MTsetmathskips{j}{5mu}{3mu} as above:
From input
\MTsetmathskips{j}{5mu}{3mu}
$ij, ijk$
$\Mathnormal{ij}, \Mathnormal{ijk}$
$i\Mathnormal{j}, i\Mathnormal{j}k$
The skips would also disappear inside \mathnormal but the latter, which is not modified by mathastext (cf its option defaultalphabets), is not configured to use the EB Garamond 12 Italic.
LuaLaTeX version:
LuaLaTeX version:
