If you want to set the UART to 9-bit mode, you will have to set the
UCSZ02 bit of the _ucsrb register to 1 in
HardwareSerial::begin().
For example:
void HardwareSerial::begin(unsigned long baud, byte config)
{
// Mostly unchanged... but by the end:
sbi(*_ucsrb, RXCIE0);
cbi(*_ucsrb, UDRIE0);
sbi(*_ucsrb, UCSZ02); // set 9-bit data mode
}
Notice that you do not need to modify the Arduino core library for this:
you could set that bit in your setup(), right after calling
Serial.begin().
Then, in the HardwareSerial::_rx_complete_irq() interrupt
handler,
you will have to read the 9th bit as the RXB80 bit in UCSR0B, before
reading the low byte from _udr (emphasis in the datasheet). For
example, you could replace the line unsigned char c = *_udr; by:
bool is_address = UCSR0B & _BV(RXB80); // read 9th bit first
unsigned char c = *_udr; // then the other bits
if (is_address) {
do_something_with_address(c);
return;
}
where do_something_with_address() is the stuff you wanted to do in
another interrupt (no need for that, you are already in interrupt
context).
It is worth noting that, with an Arduino Uno, you will not be able to do all this while sending data to the PC. This is because the communication with the PC uses the only UART of the ATmega328P.