\mathchoice aims to solve the problem of defining commands for math mode that behave differently in the four main math styles, that is, display, text, script and scriptscript. I'll gloss over the four secondary (or “cramped”) styles, which are not of a concern with \mathchoice.
When TeX is in a display, it uses display style; in an inline formula, it uses text style. But subscripts or superscripts use, at the first level, script style, and from the second level up, scriptscript style.
However, when typesetting a fraction in display style, TeX uses text style for the numerator and the denominator. But it will use script style when the fraction starts out in text style.
Let's make an example. In amsmath there is \iint that typesets a double integral; it is built by typesetting two integral signs, but they must be closer to each other than what would result by \int\int. However, the amount of back up must be bigger in display style than in the other styles, so amsmath uses
\mkern-7mu\mathchoice{\mkern-2mu}{}{}{}
which means that the back up will be -9mu in display style, but only -7mu in the other styles.
Another important example is \text@, which is amsmath internal for \text:
\def\text@#1{{\mathchoice
{\textdef@\displaystyle\f@size{#1}}%
{\textdef@\textstyle\f@size{\firstchoice@false #1}}%
{\textdef@\textstyle\sf@size{\firstchoice@false #1}}%
{\textdef@\textstyle \ssf@size{\firstchoice@false #1}}%
\check@mathfonts
}%
}
Apart from minor details, this achieves the purpose by typesetting four boxes using different fonts: \f@size is LaTeX internal for the current font size, \sf@size and \ssf@size are the internals for the font sizes corresponding to sub/superscripts of first and second level. When finally TeX makes a decision to what style the subformula containing \text is to be typeset, it will use one of the four boxes.
Those boxes are constructed before TeX know which one it has to use, because of what's perhaps the biggest flaw in TeX's design, which is how it manages fractions. But describing this would take too far.
The primitive \mathchoice has four arguments that should bybe math material; they are typeset in the corresponding style and stored away:
\mathchoice{<material for display style>}
{<material for text style>}
{<material for script style>}
{<material for scriptscript style>}
(in the definition of \text@ we see \textdef@ that calls \hbox to switch back to text mode).
The most frequent use of \mathchoice is through \mathpalette, of which you can find a description in The mysteries of \mathpalette
