1
$\begingroup$

This question is purely to satisfy my curiosity - I'm not attempting to implement my own encryption, I'm just curious.

Let's say Alice and Bob establish a communication channel, and the first thing they do is authenticate via public key/asymmetric encryption. Alice sends Bob an encrypted "handshake" packet, which is encrypted with Bob's RSA public key. Only Bob can decrypt it, as only he has the private key. Bob then does the same (sends Alice an encrypted packet, with Alice's RSA public key). Assuming all goes well, authentication is complete.

After authentication, the two need to generate a shared secret, which will be used as a key for symmetric encryption, to encrypt all further communication between the two. They use some form of the Diffie Hellman algorithm to come to the same shared secret, without transmitting the secret over the communication channel. To do this they need to exchange their DH public keys.

My question is, would it be a bad idea to exchange the DH public keys in the encrypted handshake packets that Alice and Bob sent to each other, during the authentication step? Those packets are encrypted, and they need to be sent, for purposes of authentication, so why not include the DH public keys in those encrypted packets? I understand that there is no need to encrypt the DH public keys, but can it hurt to do so? It would save some time as it would mean fewer packets, and virtually zero unencrypted packets would need to be sent throughout the whole session.

Does this approach mean losing perfect forward secrecy? I don't see how that could be the case. Yes, the confidentiality of the session would be partly tied to the long-term RSA key, but only in the sense that the DH public keys would be exposed if the RSA key was compromised. But that's no different to the approach of sending the DH public keys unencrypted, right? It's two lines of defense instead of one.

Another way to phrase my question would be: is there any reason to favor exchanging DH public keys via unencrypted packets? If you could choose either option, is there any security-related reason you would choose to expose them over keeping them secret?

I apologise in advance if this question enrages you. Again, I'm not actually implementing this, I just want to understand if/why this is a bad idea.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

What's actually needed, is to know the secret established is authenticated. A typical handshake protocol would ensure both confidentiality and authenticity.

What's desired is probably not to "encrypt" DH public keys, but to authenticate them, and forward secrecy is achieved through ephemeral key exchange.

No reasonable person can be enraged by a keen learner. I would suggest reading up the SSL/TLS protocol from your favorite encyclopedia (whether digital or dead-tree), which is the cornerstone of world wide web that people ought to be aware of, familiar with, or well-versed at depending on what they do.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, that makes sense. My approach does nothing to guarantee that the DH public key Bob received was actually sent by Alice. Eve could send her own DH public key to Bob, and Bob would be none the wiser. My approach doesn't authenticate the DH public keys, so my approach is flawed in that sense. Thanks for leading me to that. I will certainly look into the TLS protocol! $\endgroup$ Commented Nov 15, 2025 at 16:17