I have a Python gRPC server inside a Kubernetes cluster. The gRPC server implements a Ping method.
The server is load-balanced through Ingress Nginx, using the proper configuration (based on the official docs recommendations).
The server replies correctly to the request.
However, the Nginx logs show that a new HTTP/2 connection is initiated for each RPC call, despite re-using the same channel and the same stub.
NGINX logs
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:46 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:46 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:46 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:46 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:46 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:46 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:46 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:46 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:46 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:47 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:47 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:47 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:47 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
│ ingress-nginx-controller-[...] 9.x.x.120 - - [07/Sep/2024:15:52:47 +0000] "POST /My.gRPC.Api/Ping HTTP/2.0" 200 [...]
Client code
The gRPC code that produced the above logs is the following:
import grpc
from my_lib.protos import api_pb2, api_pb2_grpc
host = "grpc.example.com"
credentials = grpc.ssl_channel_credentials()
channel = grpc.secure_channel(host, credentials)
stub = api_pb2_grpc.MyApiStub(channel=channel)
request = api_pb2.PingRequest(echo="ping")
for _ in range(15):
stub.Ping(request=request)
Server code
The server is also implemented in Python.
import grpc
from my_lib.protos import api_pb2_grpc
class Server(api_pb2_grpc.MyApiServicer):
async def Ping(self, request, context):
return api_pb2.PingReply(echo=request.echo)
async def start_server():
# I tried adding the following option, but no luck.
options = [("grpc.max_concurrent_streams", 100)]
server = grpc.aio.server(options=options)
api_pb2_grpc.add_MyApi_to_server(Server(), server)
# TLS termination is done by NGINX, traffic between NGINX
# and this container is unencrypted
server.add_insecure_port("[::]:5050")
await server.start()
asyncio.run(start_server())
Question
I read everywhere that the HTTP/2 connection is expected to be re-used, and that multiple RPC calls should use the same TCP connection.
Why does a new HTTP connection is opened for each new call in my case ? What am I missing?
Resources I've read (among dozens of others):
