A serial mouse is designed to connect to a PC's true serial port - it's an actual RS-232 device using ±5V. As such it won't connect directly to the Arduino's RX/TX pins, you'll have to go through an RS-232 transceiver chip.
It also gets its power from the RTS line, but I never knew what the current draw of one of those things was - be careful trying to power it from the Arduino!
Different mouse manufacturers used different protocols. The original standard used a three-byte protocol at 1,200 bps 7N1. The encoding had the following properties:
- The leading bit of the first byte of the packet was set - all future bytes had the leading bit cleared.
- The first byte had the buttons' states, and the most significant bits of the X and Y deltas;
- The X delta was encoded in the second byte;
- The Y delta was encoded in the third byte.
Other manufacturers added things like extra buttons and scroll wheels that didn't fit in the packet structure. So they modified the protocol, but stuck with some features of the original:
- The data rate was (usually) still 1,200 bps, although sometimes they used 7N2 or 8N1 instead.
- The leading bit was still used to indicate start of packet;
- The X, Y (and Z) wheels still indicated deltas.