The role of the PHP tags <?php and ?> are to tell PHP which of the code it should parse. This is useful because you are then able to mix HTML and PHP, as you've already discovered.
An additional open tag also exists: <?=, this tag is a shortcut for <?php echo and is useful in your example, where all you are doing is outputting the value from a variable. It uses the same closing tag: ?>
There is an interesting thread about PHP tags and performance for you, right here: Opening/closing tags & performance?Opening/closing tags & performance?
Read more about the PHP tags on php.net: http://php.net/manual/en/language.basic-syntax.phptags.php