keytool error: java.io.IOException: Invalid keystore format

Question

I have a 3-node ELK stack (Elasticsearch v7.17). After a reboot, the Kibana web interface reports an error "Kibana server is not ready yet".

The SSL certs were expired, so I re-created them (for the ELK CA, all 3 nodes, Kibana, and Logstash). However, the error persists, and /var/log/kibana/kibana.log reports an error

{"type":"log","@timestamp":"2023-03-29T17:19:39+02:00","tags":["error","elasticsearch-service"],"pid":8271,"message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}

The command /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive -v results in this output:

Running with configuration path: /etc/elasticsearch

Testing if bootstrap password is valid for http://10.0.0.1:9200/_security/_authenticate?pretty
{
  "username" : "elastic",
  "roles" : [
    "superuser"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "_reserved" : true
  },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  },
  "lookup_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  },
  "authentication_type" : "realm"
}


Checking cluster health: http://10.0.0.1:9200/_cluster/health?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}


Failed to determine the health of the cluster running at http://10.0.0.1:9200
Unexpected response code [503] from calling GET http://10.0.0.1:9200/_cluster/health?pretty
Cause: master_not_discovered_exception

The Elasticsearch log say:

[2023-03-30T13:50:58,432][WARN ][o.e.d.PeerFinder         ] [node1] address [10.0.0.2:9300], node [null], requesting [false] connection failed: [][10.0.0.2:9300] general node connection failure: handshake failed because connection reset
[2023-03-30T13:50:58,432][WARN ][o.e.t.TcpTransport       ] [node1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.0.0.1:60126, remoteAddress=node2.example.org/10.0.0.2:9300, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

No password was changed. The problem appears to be with the new SSL certificates. Therefore, I have created a new keystore via the command

/usr/share/elasticsearch/bin/elasticsearch-keystore create

and I'm trying to add the CA certificate (and then others) to it:

keytool -importcert -trustcacerts -noprompt -keystore /etc/elasticsearch/elasticsearch.keystore -file /etc/elasticsearch/certs/ca.crt

However, I get the following error:

keytool error: java.io.IOException: Invalid keystore format

I have converted the CA cert into PKCS12 and tried to import it in such format (ca.p12), since the keystore is defined as of type PKCS12 in my config, but I get the same error.

What's wrong?

Excerpts of the /etc/elasticsearch/elasticsearch.yml file:

xpack.security.transport.ssl.keystore.path: elasticsearch.keystore
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.truststore.path: elasticsearch.keystore
xpack.security.transport.ssl.truststore.type: PKCS12
xpack.security.transport.ssl.verification_mode: certificate

Answer 1

Your Elasticsearch cluster is not properly set up (master_not_discovered_exception), so Kibana won't be able to use it. Because of that, Kibana cannot complete its startup.

The Elasticsearch log snippet says (split to multiple lines at \ for clarity):

io.netty.handler.codec.DecoderException: \
javax.net.ssl.SSLHandshakeException: \
PKIX path validation failed: \ 
java.security.cert.CertPathValidatorException: \ 
Path does not chain with any of the trust anchors

So the software seems to be unable to find a certification path between the currently trusted root CA certificates and the server certificate it is supposed to be using. Did you miss an intermediate CA certificate between the root CA and your server certificate?

keytool error: java.io.IOException: Invalid keystore format

The keytool command is used to manipulate Java's generic keystore files (PKCS12 and JKS formats), but it seems Elasticsearch uses its own keystore format, which can also contain arbitrary Elasticsearch settings and authentication key files in JSON format. There should be a dedicated elasticsearch-keystore tool for it:

https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-keystore.html