Return to Answer

1. The bounding box of % is not a square. Additional margins can be added to increase the frame to a square. Or the symbol can be resized. In this case the symbol is not far from a square, thus a little distorting does hurt less than different inner margins.

2. A scale factor for the rule width adaptation is calculated by comparing the height of the symbol in normal size with the height of the symbol in the current size.

3. Rounded corners can easily be made with tikz' line join=round.

4. Shrinking and moving a box can be easily done with \resizebox, \scalebox and \raisebox. The dimensions of the box are available inside the arguments of these commands through \width, \height, ...

5. Getting rid of the side bearings is the tough part, because TeX does not know the size of the side bearings. It uses only the official glyph bounding box that includes the side bearings.

LuaTeX could be tried to look into the internal font data. With some luck, the size of the side bearings can be obtained.

1. The bounding box of % is not a square. Additional margins can be added to increase the frame to a square. Or the symbol can be resized. In this case the symbol is not far from a square, thus a little distorting does hurt less than different inner margins.

2. A scale factor for the rule width adaptation is calculated by comparing the height of the symbol in normal size with the height of the symbol in the current size.

3. Rounded corners can easily be made with tikz' line join=round.

4. Shrinking and moving a box can be easily done with \resizebox, \scalebox and \raisebox. The dimensions of the box are available inside the arguments of these commands through \width, \height, ...

5. Getting rid of the side bearings is the tough part, because TeX does not know the size of the side bearings. It uses only the official glyph bounding box that includes the side bearings.

   LuaTeX could be tried to look into the internal font data. With some luck, the size of the side bearings can be obtained.

This answer is a continuation of tohecz' answer.

This answer is a continuation of tohecz' answer.

This answer is a continuation of the tohecz' answer.

\documentclass{article}
\usepackage{graphicx}
\usepackage{tikz}

\makeatletter
\newcommand*{\squareboxed@}[5]{%
  \sbox0{$#1#2\m@th$}%
  % remove side bearings
  \sbox0{%
    \kern-#4\wd0 %
    \unhcopy0 %
    \kern-#5\wd0 %
  }%
  % resize to square
  \dimen@=\dimexpr\ht0+\dp0-\wd0\relax
  \ifdim\dimen@=\z@
  \else
    \sbox0{\resizebox{\totalheight}{\height}{\unhcopy0}}%
  \fi
  % shrink and put on math axis
  \sbox2{$#1\vcenter{}$}%
  \sbox0{%
    \raisebox{\ht2}{%
      \scalebox{#3}{%
        \raisebox{.5\dimexpr\dp0-\ht0\relax}{\unhcopy0}%
      }%
    }%
  }%
  % scale \fboxrule
  \sbox2{$1$}%
  \sbox4{$#11$}%
  \fboxrule=\dimexpr\fboxrule*\ht4/\ht2\relax
  % add inner sep
  \fboxsep=.5\dimexpr#4\wd0+#5\wd0\relax
  \kern\fboxsep
  \sbox2{$#1X$}%
  \resizebox{!}{\ht2}{%
    \raisebox{\depth}{%
      \tikz[
        inner sep=\fboxsep,
        line width=\fboxrule,
        line join=round,
        baseline=(X.base),
      ]
        \node[draw](X){\copy0}%
      ;%
    }%
  }%
  \kern\fboxsep
}
\newcommand*{\squareboxed}[8]{%
  \mathchoice
    {\squareboxed@\displaystyle{#1}{#2}{#3}{#4}}%
    {\squareboxed@\textstyle{#1}{#2}{#3}{#4}}%
    {\squareboxed@\scriptstyle{#1}{#2}{#5}{#6}}%
    {\squareboxed@\scriptscriptstyle{#1}{#2}{#7}{#8}}%
}

\newcommand*{\boxmod}{%
  \mathbin{%
    \squareboxed
      {\%}   {.8}
      {.065} {.07}
      {.07}  {.075}
      {.085} {.09}%
  }%
}
\makeatother

\begin{document}
  \[
    X \boxmod Y
    \stackrel{%
      \stackrel{X\boxmod Y}{i\boxmod r} 
    }{=}
    i \boxmod r   
  \]
\end{document}

This answer is a continuation of tohecz' answer.

\documentclass{article}
\usepackage{graphicx}
\usepackage{tikz}

\makeatletter
\newcommand*{\squareboxed@}[5]{%
  \sbox0{$#1#2\m@th$}%
  % remove side bearings
  \sbox0{%
    \kern-#4\wd0 %
    \unhcopy0 %
    \kern-#5\wd0 %
  }%
  % resize to square
  \dimen@=\dimexpr\ht0+\dp0-\wd0\relax
  \ifdim\dimen@=\z@
  \else
    \sbox0{\resizebox{\totalheight}{\height}{\unhcopy0}}%
  \fi
  % shrink and put on math axis
  \sbox2{$#1\vcenter{}$}%
  \sbox0{%
    \raisebox{\ht2}{%
      \scalebox{#3}{%
        \raisebox{.5\dimexpr\dp0-\ht0\relax}{\unhcopy0}%
      }%
    }%
  }%
  % scale \fboxrule
  \sbox2{$1$}%
  \sbox4{$#11$}%
  \fboxrule=\dimexpr\fboxrule*\ht4/\ht2\relax
  % add inner sep
  \fboxsep=.5\dimexpr#4\wd0+#5\wd0\relax
  \kern\fboxsep
  \sbox2{$#1X$}%
  \resizebox{!}{\ht2}{%
    \raisebox{\depth}{%
      \tikz[
        inner sep=\fboxsep,
        line width=\fboxrule,
        line join=round,
        baseline=(X.base),
      ]
        \node[draw](X){\copy0}%
      ;%
    }%
  }%
  \kern\fboxsep
}
\newcommand*{\squareboxed}[8]{%
  \mathchoice
    {\squareboxed@\displaystyle{#1}{#2}{#3}{#4}}%
    {\squareboxed@\textstyle{#1}{#2}{#3}{#4}}%
    {\squareboxed@\scriptstyle{#1}{#2}{#5}{#6}}%
    {\squareboxed@\scriptscriptstyle{#1}{#2}{#7}{#8}}%
}

\newcommand*{\boxmod}{%
  \mathbin{%
    \squareboxed
      {\%}   {.8}
      {.065} {.07}
      {.07}  {.075}
      {.085} {.09}%
  }%
}
\makeatother

\begin{document} 

  \noindent
  $ X \boxmod Y$\\
  $ r \boxmod i ^ {r \boxmod i ^ {r \boxmod i}}$\\
  $ n - (n \boxmod m)$

\end{document}