Volume 15 of the MTS manual series, "FORMAT and TEXT360", can be found e.g. at Bitsavers.
The manual uses / instead of |, but otherwise the description of the "incantations" seems to be at least close. E.g. /p/ starts a new paragraph, /H.../ changes column margins, etc.
Edit: According to this document, 'FMT' and 'FORMAT' are indeed related:
It is called FMT, and
is very similar to John Ehrman's FORMAT, but is a
complete rewrite (In System/360 Assembler Language
not FORTRAN) and is such more powerful.
This mentions the "FMT Reference manual", by "J. Pearkins, U. of Alberta", but I couldn't find this one online.
It also mentions another program called 'TYPESET'
Incldently, I know of (but nothing about) a
program called TYPESET, which is advertised as an
"extended version" of FMT. Writeups are (or, at
least, a year ago, were) available free of charge from: Grant Crawford, Computer Services, University
of Alberta, Edmonton, Alberta.
and "Typeset User's Guide (Preliminary Draft)", by "G. Crawford, U. of Alberta.", which I couldn't find online, either.
But maybe the 'FORMAT' commands can get you started.
BTW, it looks like you can run MTS on your own computer with help of the Hercules emulator. The original http://archive.michigan-terminal-system.org/ seems to be gone, but it can be still viewed via archive.org, and the distribution is also on bitsavers. There's also a walkthrough written 2014.
