storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexandre Vermeerbergen <avermeerber...@gmail.com>
Subject How to setup Kafka authentication on storm-kafka-monitor ?
Date Mon, 03 Jun 2019 08:52:06 GMT
Hello,

I am trying to switch my Storm topologies based on Kafka spout from
authenticated & unencrypted Kafka brokers to SSL + Authenticated Kafka
Brokers.

So I have changed my Kafka spouts build code to something like this
(actial user & password have been changed to 'foo' and 'bar'):

kafkaSpoutConfig =
KafkaSpoutConfig.builder(supConfig.kafka_broker_hosts_str, "myTopic")
                    .setProp(ConsumerConfig.GROUP_ID_CONFIG, "myConsumer")
                    .setProp("security.protocol", "SASL_SSL")
                    .setProp("sasl.jaas.config",
"org.apache.kafka.common.security.scram.ScramLoginModule required
username=\"foo\" password=\"bar\";")
                    .setProp("sasl.mechanism", "SCRAM-SHA-512")

.setProp(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "")
                    .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.LATEST)
                    .setRecordTranslator(new PodSyncPodsKafkaRecordTranslator())
                    .build();


My problem is that in Storm UI, I see that the usual Kafka Lag
monitoring fails with these errors:

Topology spouts lag error

IdTypeMessage
eventFromAdminTopicKAFKAUnable to get offset lags for kafka. Reason:
org.apache.kafka.shaded.common.KafkaException: Failed to construct
kafka consumer at
org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:703)
at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:584)
at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:566)
at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:230)
at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:144)
Caused by: org.apache.kafka.shaded.common.KafkaException:
java.lang.IllegalArgumentException: Could not find a 'KafkaClient'
entry in the JAAS configuration. System property
'java.security.auth.login.config' is not set at
org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
at org.apache.kafka.shaded.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
at org.apache.kafka.shaded.clients.ClientUtils.createChannelBuilder(ClientUtils.java:85)
at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644)
... 4 more Caused by: java.lang.IllegalArgumentException: Could not
find a 'KafkaClient' entry in the JAAS configuration. System property
'java.security.auth.login.config' is not set at
org.apache.kafka.shaded.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:65)
at org.apache.kafka.shaded.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
at org.apache.kafka.shaded.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
at org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
... 7 more

Question:

I guess that this issue comes from the fact that storm-kafka-monitor
is not aware of the security setup which have been applied to the
Spout.
=> How can I configure storm-kafka-monitor to pass it the Kafka
consumer properties that will allow it to query the Kafka lag without
failing on Kafka authentication?

I have not found a clue about this in documentation...

Also, is it normal that Kafka spouts is not automatically inherited by
storm-kafka-monitor?

Kind regards,
Alexandre Vermeerbergen

Mime
View raw message