the integration of NGINX and HSM in TLS offloading

describe that > During the TLS handshake, when the client sends the encrypted pre-master secret, NGINX uses PKCS#11 to pass this to the HSM.

You seem to refer here to the RSA key exchange, which is considered obsolete for long time since it does not provide forward secrecy. Unfortunately AWS still used it in their documentation instead of modern key exchange. Note that RSA key exchange is not only considered obsolete, but it is simply no longer available with TLS 1.3, so please regard the relevant AWS documentation as broken.

Does the HSM participate in encryption/decryption of the TLS session data? AWS CloudHSM article did not elaborate if HSM gets involve after the handshake process ends

"TLS offloading" is a very generic term, which just means that some or all part of TLS is offloaded to somewhere else. Sometimes it describes offloading encryption and decryption from user space to kernel or an hardware accelerator. Sometimes it describes an reverse proxy which terminates TLS and thus offloads TLS from the web application.

The documentation in AWS is vague (or even misleading) here in that it claims "... establish an HTTPS session with each client. This process involves a lot of computation for web servers, but you can offload some of this to your AWS CloudHSM cluster, which is referred to as SSL acceleration. Offloading reduces the computational burden on your web servers and provides extra security by storing servers’ private keys in HSMs".

In reality from the documentation of the configuration it can be seen that it is only about having the private key for the server certificate is in the HSM, so that any operations involving the private key must be done by the HSM. This specifically means the signature operation to create a CertificateVerify message. Anything not involving the private key is not handled by the CloudHSM, i.e. key exchange, session handling and encryption/decryption - contrary what AWS might suggest with mentioning "SSL acceleration".

If not, then does NGINX handle symmetric encryption/decryption, and where does it store the session key?

Both key exchange and encryption/decryption are handled by nginx with the session key stored in memory.

Is the communication between HSM and nginx server encrypted?

To cite from another AWS documentation: "Communication between the AWS CloudHSM client and the HSM in your cluster is encrypted from end to end."