Kafka SaslAuthenticationException Occuring on Ad-hoc basis for SASL_SSL Protocol

Question

I am running some java 8 Kafka Applications, some of them Kafka streams, the others are plain producer/consumer.

For each of them, there are no functionality issues, they run fine for most of the time.

However, for each of them, I am getting adhoc occurences of SaslAuthenticationException. As they occur once every couple of weeks or so, I am not sure how to replicate/deduce the root cause:

Error:

Failed to create channel due to 
org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.KafkaException: Principal could not be determined from Subject, this may be a transient failure due to Kerberos re-login

Stack Trace:

Failed to create channel due to 
org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.KafkaException: Principal could not be determined from Subject, this may be a transient failure due to Kerberos re-login
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.firstPrincipal(SaslClientAuthenticator.java:441) ~[kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.<init>(SaslClientAuthenticator.java:135) ~[kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.common.network.SaslChannelBuilder.buildClientAuthenticator(SaslChannelBuilder.java:244) ~[kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:194) ~[kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:289) [kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:280) [kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.common.network.Selector.connect(Selector.java:215) [kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:864) [kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.clients.NetworkClient.access$700(NetworkClient.java:64) [kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1035) [kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:920) [kafka-clients-2.0.1.jar!/:na]
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:508) [kafka-clients-2.0.1.jar!/:na]

This is how I am providing my JAAS Kerberos Authentication to Kafka (In my config file I have provided information like the kdc, realm, keytab, principal):

@Value("${kafka.sasl.kerberos.kdc}")
private String kdc;
@Value("${kafka.sasl.kerberos.realm}")
private String realm;
@Value("${kafka.sasl.kerberos.keytab}")
private Resource keytab;
@Value("${kafka.sasl.kerberos.principal}")
private String principal;

@Bean
public InMemoryConfiguration kafkaOpts() throws IOException {

    System.setProperty("java.security.krb5.kdc", kdc);
    System.setProperty("java.security.krb5.realm", realm);

    Map<String, Object> options = new HashMap<>();
    options.put("keyTab", copyResourceToTempFile(keytab, ".keytab").toString());
    options.put("principal", principal);
    options.put("useKeyTab","true");
    options.put("storeKey","true");
    AppConfigurationEntry kafkaClientConfig = new AppConfigurationEntry(
            "com.sun.security.auth.module.Krb5LoginModule", LoginModuleControlFlag.REQUIRED, options);
    Map<String, AppConfigurationEntry[]> jaasConfigEntries = new HashMap<>();
    jaasConfigEntries.put("KafkaClient", new AppConfigurationEntry[] {kafkaClientConfig});
    InMemoryConfiguration jaasConfig = new InMemoryConfiguration(jaasConfigEntries);
    javax.security.auth.login.Configuration.setConfiguration(jaasConfig);
    return jaasConfig;
}

public static Path copyResourceToTempFile(Resource resource, String extension) {

    try (InputStream in = resource.getInputStream()) {
        Path tempFile = Files.createTempFile("spring-boot-", extension);
        Files.copy(in, tempFile, StandardCopyOption.REPLACE_EXISTING);
        return tempFile;
    } catch (IOException e) {
        log.error("Error creating resource to file",e);
        return null;
    }

}

adiga · Accepted Answer · 2019-06-24 07:48:25Z

2

We also faced same issue. After connecting with kafka, suddenly we loose connection and application retry to login but it fails. But We update jaas auth config files with following configuration:

com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true;

It is perfectly working for me.

edited Jun 24, 2019 at 7:48

adiga

35.3k9 gold badges65 silver badges87 bronze badges

answered Jun 24, 2019 at 7:42

Rajeev Rathor

1,94225 silver badges23 bronze badges

1

I think this fetches from the TGT cache in the /tmp/krb5cc_uid folder and works only for the duration when the TGT in your server is valid, but do you need another process to refresh the TGT periodically? For my case, I am using keytab to let the program refresh the TGT automatically. Actually, we managed to solve the issue by pointing to a static keytab file instead of using a temp keytab file. The latter was the cause of the issue, as it gets wiped off by other programs cleaning up the tmp folder, resulting in the program not being able to use the temp keytab file to refresh the TGT.
– chaooder
Commented Jun 25, 2019 at 0:48

Add a comment |

Collectives™ on Stack Overflow

Kafka SaslAuthenticationException Occuring on Ad-hoc basis for SASL_SSL Protocol

1 Answer 1

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Linked

Related