`\mathchoice` aims to solve the problem of defining commands for math mode that behave differently in the four main math styles, that is, *display*, *text*, *script* and *scriptscript*. I'll gloss over the four secondary (or “cramped”) styles, which are not of a concern with `\mathchoice`.

When TeX is in a display it uses display style; in an inline formula it uses text style. But subscripts or superscripts use, at the first level, script style, and from the second level up, scriptscript style.

However, when typesetting a fraction in display style, TeX uses text style for the numerator and the denominator. But it will use script style when the fraction starts out in text style.

Let's make an example. In `amsmath` there is `\iint` that typesets a double integral; it is built by typesetting two integral signs, but they must be closer to each other than what would result by `\int\int`. However, the amount of back up must be bigger in display style than in the other styles, so `amsmath` uses

    \mkern-7mu\mathchoice{\mkern-2mu}{}{}{}

which means that the back up will be -9mu in display style, but only -7mu in the other styles.

Another important example is `\text@`, which is `amsmath` internal for `\text`:

    \def\text@#1{{\mathchoice
      {\textdef@\displaystyle\f@size{#1}}%
      {\textdef@\textstyle\f@size{\firstchoice@false #1}}%
      {\textdef@\textstyle\sf@size{\firstchoice@false #1}}%
      {\textdef@\textstyle \ssf@size{\firstchoice@false #1}}%
      \check@mathfonts
      }%
    }

Apart from minor details, this achieves the purpose by typesetting four boxes using different fonts: `\f@size` is LaTeX internal for the current font size, `\sf@size` and `\ssf@size` are the internals for the font sizes corresponding to sub/superscripts of first and second level. When finally TeX makes a decision to what style the subformula containing `\text` is to be typeset, it will use one of the four boxes.

Those boxes are constructed *before* TeX know which one it has to use, because of what's perhaps the biggest flaw in TeX's design, which is how it manages fractions. But describing this would take too far.

The primitive `\mathchoice` has four arguments that should by math material; they are typeset in the corresponding style and stored away:

    \mathchoice{<material for display style>}
               {<material for text style>}
               {<material for script style>}
               {<material for scriptscript style>}

(in the definition of `\text@` we see `\textdef@` that calls `\hbox` to switch back to text mode).

The most frequent use of `\mathchoice` is through `\mathpalette`, of which you can find a description in https://tex.stackexchange.com/questions/34393/the-mysteries-of-mathpalette