End-to-end encryption is hard, even without the guest account requirement!
With that said, fundamentally, your need isn't that complicated. Some user wants to send a message to a guest. The user's client (browser-side script, if it's a webapp) generates a symmetric encryption key (no reason to use public keys here) and encrypts the message(s) that the user sends. The encrypted messages are stored on the server, along with some identification (potentially the email address of the intended recipient, though that requires storing some plain-text PII so maybe you use an opaque identifier). The client also generates a URL, containing the key as part of the URL fragment (and, if relevant, the opaque identifier as a path or query parameter). This URL can be used to retrieve the encrypted messages and (via the fragment, which is not sent to the server) decrypt them.
The URL gets sent to the recipient (guest). You have a choice to make here: how to get that link from the user's client to the guest recipient's inbox? There are broadly speaking three ways:
- Send the URL to the server, and the server sends the email. This violates the zero-trust property, as it requires trusting that the server does not store the URL (or even just the key portion), or use it while it's transiently passing through the system, or keep it in a "sent messages" log on the mail server, or whatever.
- Tell the user to send the link themselves (that is, copy it to the clipboard and paste it into their own email client, or chat program, or even write it on a letter and send it via snail-mail). Not a great user experience, but hey, it (mostly) preserves zero trust... as long as you aren't using a webapp, which is basically incompatible with zero-trust (the user and recipient always need to trust that the server isn't sending them malicious script that steals the plaintext; in theory you could manually validate this every time since even minified client JS is technically readable, but in practice, ha, no).
- Have the client app send the message with minimal user interaction but without relaying off the server. For a webapp, the closest you can probably get to that is to have the user click a
mailto: URI, and hope that the user's browser is configured to open those in the right email program or webapp and that the email client (web or otherwise) parses it as expected (and that the user then clicks "Send"). For a desktop/mobile app, you could in theory have the app connect to an SMTP server directly (probably the user's server that they have to supply credentials to, which is likely a non-starter) but in reality you'd probably use a "share contract" to send the URL to a messaging app of some sort via OS-provided functionality for passing data and intent between apps.
I'm leaving out a ton of detail here, mind you. You want integrity protection, not just encryption, on the ciphertext; this should probably be done using authenticated encryption. You can also sign messages if you want them to provably come from that specific user, but sometimes sender privacy / message repudiation is a desirable property instead. You want some way to expire the links, used or unused, and probably expire them sooner if used... but you probably do not want them to expire immediately upon first use, otherwise any program that pre-fetches URLs (e.g. to check them through a security scanner, display a preview image of the resulting page, or cache them for faster loading later) might expire the link prematurely. The system as described only handles sending to guests, not guests starting a conversation or replying to a message; both are possible but it gets more complicated. This also is primarily intended for a single recipient; if you want to send the same message to multiple people, without duplicating the same plaintext across a bunch of ciphertexts, then you potentially do want to use a hybrid cryptosystem where you send the guest a private key; elliptic curve keys are short enough that this is viable but it's still adding complexity again.
Perhaps most importantly, you need to think about your threat model here. The whole question of what you're really trying to achieve, security-wise, against whom. End-to-end encryption without zero trust means little, but as mentioned above, zero trust in webapps is basically unfeasible. You'd also run into problems when encrypting to other registered users, unless you have some way to verify - in the client - that the key (probably public key) you're encrypting to belongs to that user and nobody else (like the server) has its private key, and more complexity still if you want forward secrecy.
These are all solvable problems. Take a look at Signal and its ratcheting key exchange if you want to see an open source example of a widely-used zero-trust end-to-end encrypted messaging platform with support for asynchronous transmission of large messages, complete with forward secrecy. It's a beautiful design. But it's not something a novice should try to implement, and frankly, most people don't have a good reason to do so.
