I think using the FilesystemIterator should be the fastest and easiest way:
// PHP 5 >= 5.3.0
$iterator = new \FilesystemIterator($dir);
$isDirEmpty = !$iterator->valid();
Or using class member access on instantiation:
// PHP 5 >= 5.4.0
$isDirEmpty = !(new \FilesystemIterator($dir))->valid();
This works because a new FilesystemIterator will initially point to the first file in the folder - if there are no files in the folder, valid() will return false. (see documentation here.)
As pointed out by abdulmanov.ilmir, optionally check if the directory exists before using the FileSystemIterator because otherwise it'll throw an UnexpectedValueException.