High frequency authentication requests between GCP Dataflow and Managed Kafka

Question

I'm experiencing an issue with frequent authentication requests between Google Cloud Dataflow and GCP Managed Kafka. Each Dataflow streaming job is making approximately 150 authentication requests per minute to Kafka, which is causing problems when running multiple jobs.Current SetupInfrastructure Configuration

• Kafka partitions per topic: 4

• Dataflow worker: 1 instance (n2-standard-4, 4vCores)

Authentication and Consumer Configuration

    // Kafka Authentication Configuration
    "security.protocol"                 to "SASL_SSL",
    "sasl.mechanism"                    to "PLAIN",
    "sasl.login.callback.handler.class" to             "com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler",
"sasl.jaas.config"                  to "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;"

// Consumer Configuration
val configMap = mutableMapOf(
    "group.id" to consumerGroupName,
    "enable.auto.commit" to "false",
    "auto.commit.interval.ms" to "5000"
)

Issue Description

When running 9-10 streaming applications simultaneously, we exceed the Kafka broker's authentication limit of 1,200 requests per minute, resulting in the following error:

Quota exceeded for quota metric 'Cluster authenticate connection requests' and limit 'Cluster authenticate connection requests per minute per region' of service 'managedkafka.googleapis.com' for consumer

The frequency of authentication requests (150 per minute per job) seems unusually high. Is there a way to reduce the number of authentication requests while maintaining the functionality of the Dataflow jobs?

Here are the configuration options I've modified:

1. Adjusted session and heartbeat settings:

   • Increased session.timeout.ms

   • Modified heartbeat.interval.ms

   • Changed connection-related timeout parameters
     Result: Issue persisted with no improvement in authentication request frequency

2. Changed authentication mechanism:

   • Switched from OAUTHBEARER to PLAIN authentication mechanism

   • Modified related SASL configurations
     Result: No reduction in authentication requests observed

3. Modified SASL refresh parameters:

   • Adjusted sasl.login.refresh.window.factor

   • Changed sasl.login.refresh.window.jitter

   • Modified sasl.login.refresh.min.period.seconds

   • Updated sasl.login.refresh.min.buffer.seconds
     Result: Authentication request frequency remained unchanged