According to the datasheet, the ATMega168 can only execute self-programming instructions when running code from the bootloader block.
Once the bootloader is complete and the application starts, can the application then jump back into a subroutine in the bootloader block and have it write to the flash?
Or, is it the case that once the program leaves the boot block it's not allowed to self-program again until after a reset?
well... you can always execute a software initiated reset ala exploitation of the watchdog timer... and put data in eeprom to communicate to your bootloader code as to what you wanted it to do...
http://www.cs.mun.ca/~paul/cs4723/material/atmel/avr-libc-user-manual-1.6.5/FAQ.html#faq_softreset
Looking at page 272 of the ATMega168 manual (section 6: Entering the Boot Loader Program), the first line states:
Entering the Boot Loader takes place by a jump or call from the application program. This may be initiated by a trigger such as a command received via USART, or SPI interface.
So it would seem to me that definitively you can and must be able to call the bootloader functions from application code. As long as you are executing code that resides in the Boot Loader section of memory, SPM instructions can be executed, and therefore you should be able to self-program any time your application program jumps into the boot loader code.
You can just jump to the bootloader block and everything should be fine. (just remember to move the interrupt vectors as well)
You can jump into bootloader and start reprogramming the application part, but that's probably a bad design. Consider what happens if an update fails or gets interrupted: now your app area is trashed, and since half your bootloader effectively lives there, it's probably trashed too. So you've effectively bricked the device.
You want to structure the bootloader as an independent program.
See this FAQ:
http://www.avrfreaks.net/forum/faq-c-writing-bootloader-faq?name=PNphpBB2&file=viewtopic&t=79206
