kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kaushik srinivas (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-12534) kafka-configs does not work with ssl enabled kafka broker.
Date Mon, 17 May 2021 05:53:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-12534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17345886#comment-17345886
] 

kaushik srinivas edited comment on KAFKA-12534 at 5/17/21, 5:52 AM:
--------------------------------------------------------------------

Hi,

We have tried the exact steps. Captured the commands and logs in detail. The scenario to change
the keystore password does not work still. 

sequence of steps to reproduce
 # install kafka broker by generating a CA, truststore and keystore. (password for stores:
123456)
 # re generate the keystore with a new password (1234567). Use the same old generated CA and
trust store from step1.
 # issue the dynamic reconfig command after replacing the keystore file in the specified location.
 # dynamic config command issued: 
{code:java}
./kafka-configs --bootstrap-server kafkabroker:9092 --command-config ssl.properties --entity-type
brokers --entity-name 1001 --alter --add-config 'listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
{code}
Note: listener name is ssl and is in the format specified in [https://docs.confluent.io/platform/current/kafka/dynamic-config.html#updating-ssl-keystore-of-an-existing-listener]

 # command fails with following trace 
{code:java}
Error while executing config command with args '--bootstrap-server kafkabroker:9092 --command-config
ssl.properties --entity-type brokers --entity-name 1001 --alter --add-config listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException:
Invalid config value for resource ConfigResource(type=BROKER, name='1001'): Invalid value
org.apache.kafka.common.config.ConfigException: Validation of dynamic config update of SSLFactory
failed: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore
of type JKS for configuration Invalid dynamic configuration
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
        at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:338)
        at kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:308)
        at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:85)
        at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for
resource ConfigResource(type=BROKER, name='1001'): Invalid value org.apache.kafka.common.config.ConfigException:
Validation of dynamic config update of SSLFactory failed: org.apache.kafka.common.KafkaException:
Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for configuration
Invalid dynamic configuration

{code}
Kafka broker logs the below output 

 
{code:java}
{ "timezone":"UTC", "log":{"message":"data-plane-kafka-request-handler-5 - kafka.server.AdminManager
- [Admin Manager on Broker 1001]: Invalid config value for resource ConfigResource(type=BROKER,
name='1001'): Invalid value org.apache.kafka.common.config.ConfigException: Validation of
dynamic config update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed
to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for configuration Invalid
dynamic configuration"}} {code}

 

As per docs, the CA is not supposed to be changed and we have maintained that and the CA and
trust stores are not changed. Also another observation is that, when for example the country
name in the cert generation is changed and the certificate is regenerated, the dynamic config
command works fine and we could see the ssl certs being reloaded in the kafka broker logs.

But when the keystore password is changed, things have never worked for us even after so many
attempts of retries. Can you please help in reproducing this issue and provide some detailed
steps if possible for the case where the keystore's password is being changed ? It has clearly
never worked for us, even after many attempts.


was (Author: kaushik srinivas):
Hi,

We have tried the exact steps. Captured the commands and logs in detail. The scenario to change
the keystore password does not work still. 

sequence of steps to reproduce
 # install kafka broker by generating a CA, truststore and keystore. (password for stores:
123456)
 # re generate the keystore with a new password (1234567). Use the same old generated CA and
trust store from step1.
 # issue the dynamic reconfig command after replacing the keystore file in the specified location.
 # dynamic config command issued: 
{code:java}
./kafka-configs --bootstrap-server kafkabroker:9092 --command-config ssl.properties --entity-type
brokers --entity-name 1001 --alter --add-config 'listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
{code}
Note: listener name is ssl and is in the format specified in [https://docs.confluent.io/platform/current/kafka/dynamic-config.html#updating-ssl-keystore-of-an-existing-listener]
 # command fails with following trace 
{code:java}
Error while executing config command with args '--bootstrap-server kafkabroker:9092 --command-config
ssl.properties --entity-type brokers --entity-name 1001 --alter --add-config listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException:
Invalid config value for resource ConfigResource(type=BROKER, name='1001'): Invalid value
org.apache.kafka.common.config.ConfigException: Validation of dynamic config update of SSLFactory
failed: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore
of type JKS for configuration Invalid dynamic configuration
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
        at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:338)
        at kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:308)
        at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:85)
        at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for
resource ConfigResource(type=BROKER, name='1001'): Invalid value org.apache.kafka.common.config.ConfigException:
Validation of dynamic config update of SSLFactory failed: org.apache.kafka.common.KafkaException:
Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for configuration
Invalid dynamic configuration

{code}
Kafka broker logs the below output 
{code:java}
{ "timezone":"UTC", "log":{"message":"data-plane-kafka-request-handler-5 - kafka.server.AdminManager
- [Admin Manager on Broker 1001]: Invalid config value for resource ConfigResource(type=BROKER,
name='1001'): Invalid value org.apache.kafka.common.config.ConfigException: Validation of
dynamic config update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed
to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for configuration Invalid
dynamic configuration"}}
{code}
{code:java}
 {code}

> kafka-configs does not work with ssl enabled kafka broker.
> ----------------------------------------------------------
>
>                 Key: KAFKA-12534
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12534
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 2.6.1
>            Reporter: kaushik srinivas
>            Priority: Critical
>
> We are trying to change the trust store password on the fly using the kafka-configs script
for a ssl enabled kafka broker.
> Below is the command used:
> kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name
1001 --alter --add-config 'ssl.truststore.password=xxx'
> But we see below error in the broker logs when the command is run.
> {"type":"log", "host":"kf-2-0", "level":"INFO", "neid":"kafka-cfd5ccf2af7f47868e83473408",
"system":"kafka", "time":"2021-03-23T12:14:40.055", "timezone":"UTC", "log":\{"message":"data-plane-kafka-network-thread-1002-ListenerName(SSL)-SSL-2
- org.apache.kafka.common.network.Selector - [SocketServer brokerId=1002] Failed authentication
with /127.0.0.1 (SSL handshake failed)"}}
>  How can anyone configure ssl certs for the kafka-configs script and succeed with the
ssl handshake in this case ? 
> Note : 
> We are trying with a single listener i.e SSL: 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message