I send an email to all my colleagues a few days before going on vacation (also setup an auto responder, and a similar worded one for external contacts (clients)).
It says something similar to:
I will be on annual leave from start_date and returning on the return_date. If you need me for anything urgent please contact the project_manager.
Please re-send any emails after I return. (optional line: as I will not be available via phone or email during this time)
Regards my_name.
You can then talk to your project manager and tell them, that during vacation you will be away from your phone and computer, so you wontwon't be checking any messages.
Then you can ignore all work calls. Even better, keep a dedicated work phone, then you can just turn it off during vacation.
And no, I don't see any of this as "passive-aggressive". - but why are you worried about that, considering your collages are contacting you on vacation?