Apache multiple virtual hosts same ServerName

I want to use the same exact ServerName, but load the first virtualhost block that works.

Here's how it should work:

If there is a wordpress folder in /var/www load that. Otherwise, load index.html in the /var/www directory.

Here is my config file with the 2 virtualhost blocks how I thought that would work. But it only works if I remove one or the other.

<VirtualHost *:80>
    DirectoryIndex index.php index.html
    DocumentRoot /var/www/wordpress
    ServerAdmin [email protected]
    ServerName devsite.test

    <Directory "/var/www/wordpress">
        Options FollowSymLinks
        AllowOverride All
    </Directory>

    ErrorLog /var/log/apache2/error.log
    CustomLog /var/log/apache2/access.log combined
</VirtualHost>

<VirtualHost *:80>
    DirectoryIndex index.php index.html
    DocumentRoot /var/www
    ServerAdmin [email protected]
    ServerName devsite.test

    <Directory "/var/www">
        Options FollowSymLinks
        AllowOverride All
    </Directory>

    ErrorLog /var/log/apache2/error.log
    CustomLog /var/log/apache2/access.log combined
</VirtualHost>

Thanks!