A serial mouse is designed to connect to a PC's true serial port - it's an actual RS-232 device using ±5V. As such it won't connect directly to the Arduino's RX/TX pins, you'll have to go through an RS-232 transceiver chip.
It also gets its power from the RTS line, but I never knew what the current draw of one of those things was - be careful trying to power it from the Arduino!
The actual protocol is a series of packets sent at 9,600 bps (8N1): from memory the early ones issued 5 data bytes then a line end:
- The first two bytes were the X-delta;
- The second two bytes were the Y-delta;
- The fifth byte was the different button states (one bit for each);
- Then an
<LF>(hex0x0A)?
I'll look up my old reference books...
Edit:
- Some quick Googling told me that the original protocol bears no resemblance to my memories: http://paulbourke.net/dataformats/serialmouse/
- And here's a page that describes a couple of other protocols: the last is much closer to my memory (albeit at 1,200 bps): http://www.cpcwiki.eu/index.php/Serial_RS232_Mouse