As per this comment from the OP, this is MythQuest (2001).
From Wikipedia:
MythQuest is a Canadian television series that originally aired on PBS in 2001. Produced by Mind's Eye Entertainment, it stars Meredith Henderson and Christopher Jacot as Cleo and Alex Bellows, two teens whose father Matt disappears into the Cyber Museum, a computer program that, as they discover, allows them to travel into myths by touching an artifact on the screen.
They become characters in the myths, and strive to keep the myth to its normal progression; a difficult task when confronted by Gorgos, a trickster god who appears in each myth to try to change it. Alex's initial journey into the Cyber Museum is an accident, but soon they both begin searching for their father within the myths. As the series progresses, they realize that their search is much more important than they first realized, as Gorgos is bent on wiping out entire cultures by corrupting their mythologies.
It's not a perfect match to the description, as the boy and girl are siblings rather than neighbours, and the girl is wheelchair-bound, not the boy. Also, it's a computer program rather than a book that transports them into the bodies of mythological figures.
In the first episode, "The Minotaur", the boy, Alex, is transported into the body of the Greek mythological figure of Theseus and ends up battling the Minotaur in King Minos' labyrinth. His sister, Cleo, stays in the real world, and observes what he's doing on a large, wall-mounted, flat screen monitor. They can also hear one another speak, so she helps him by providing intel on his situation.
In the third episode, "Red Wolf's Daughter", Cleo is transported into the body of a Native American girl, and discovers she can walk while in that body. Alex stays in the real world and provides intel this time.
The seventh and eighth episodes, "Isis & Osiris: Part 1" and "Isis & Osiris: Part 2", take place in ancient Egypt.
You can view the scene where Alex is transported into the myth of Theseus at around the 12:50 mark in the video below.