storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Nemuri <nhsande...@gmail.com>
Subject Re: How to setup Kafka authentication on storm-kafka-monitor ?
Date Mon, 03 Jun 2019 15:32:37 GMT
I guess you are missing the KafkaClient section in
*${STORM_CONF_DIR}/storm_jaas.conf* in UI node.

On Mon, Jun 3, 2019 at 8:32 PM Stig Rohde Døssing <stigdoessing@gmail.com>
wrote:

> There was a recent PR to partially address this at
> https://github.com/apache/storm/pull/3016, it probably doesn't fix your
> exact case, but fixing your case would probably require doing something
> similar.
>
> The reason this is broken is that storm-kafka-monitor is a workaround to
> us not having generic metrics delivery from spouts to Storm UI. The way
> storm-kafka-monitor works, is that your spout reports some of its
> configuration to Nimbus. When someone requests a Storm UI page, Nimbus then
> boots up a new JVM running storm-kafka-monitor, which just starts a
> KafkaConsumer with the config from Nimbus, checks the current consumer
> position, and exits.
>
> We want to get rid of storm-kafka-monitor eventually. I've been looking at
> it in https://issues.apache.org/jira/browse/STORM-3202, but it will
> probably take a while to get done. If you'd like to take a look at
> implementing that issue, I'd be happy to share the code (it does not
> currently compile).
>
> Den man. 3. jun. 2019 kl. 10.52 skrev Alexandre Vermeerbergen <
> avermeerbergen@gmail.com>:
>
>> Hello,
>>
>> I am trying to switch my Storm topologies based on Kafka spout from
>> authenticated & unencrypted Kafka brokers to SSL + Authenticated Kafka
>> Brokers.
>>
>> So I have changed my Kafka spouts build code to something like this
>> (actial user & password have been changed to 'foo' and 'bar'):
>>
>> kafkaSpoutConfig =
>> KafkaSpoutConfig.builder(supConfig.kafka_broker_hosts_str, "myTopic")
>>                     .setProp(ConsumerConfig.GROUP_ID_CONFIG, "myConsumer")
>>                     .setProp("security.protocol", "SASL_SSL")
>>                     .setProp("sasl.jaas.config",
>> "org.apache.kafka.common.security.scram.ScramLoginModule required
>> username=\"foo\" password=\"bar\";")
>>                     .setProp("sasl.mechanism", "SCRAM-SHA-512")
>>
>> .setProp(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "")
>>
>> .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.LATEST)
>>                     .setRecordTranslator(new
>> PodSyncPodsKafkaRecordTranslator())
>>                     .build();
>>
>>
>> My problem is that in Storm UI, I see that the usual Kafka Lag
>> monitoring fails with these errors:
>>
>> Topology spouts lag error
>>
>> IdTypeMessage
>> eventFromAdminTopicKAFKAUnable to get offset lags for kafka. Reason:
>> org.apache.kafka.shaded.common.KafkaException: Failed to construct
>> kafka consumer at
>>
>> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:703)
>> at
>> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:584)
>> at
>> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:566)
>> at
>> org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:230)
>> at
>> org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:144)
>> Caused by: org.apache.kafka.shaded.common.KafkaException:
>> java.lang.IllegalArgumentException: Could not find a 'KafkaClient'
>> entry in the JAAS configuration. System property
>> 'java.security.auth.login.config' is not set at
>>
>> org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
>> at
>> org.apache.kafka.shaded.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
>> at
>> org.apache.kafka.shaded.clients.ClientUtils.createChannelBuilder(ClientUtils.java:85)
>> at
>> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644)
>> ... 4 more Caused by: java.lang.IllegalArgumentException: Could not
>> find a 'KafkaClient' entry in the JAAS configuration. System property
>> 'java.security.auth.login.config' is not set at
>>
>> org.apache.kafka.shaded.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:65)
>> at
>> org.apache.kafka.shaded.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
>> at
>> org.apache.kafka.shaded.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
>> at
>> org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
>> ... 7 more
>>
>> Question:
>>
>> I guess that this issue comes from the fact that storm-kafka-monitor
>> is not aware of the security setup which have been applied to the
>> Spout.
>> => How can I configure storm-kafka-monitor to pass it the Kafka
>> consumer properties that will allow it to query the Kafka lag without
>> failing on Kafka authentication?
>>
>> I have not found a clue about this in documentation...
>>
>> Also, is it normal that Kafka spouts is not automatically inherited by
>> storm-kafka-monitor?
>>
>> Kind regards,
>> Alexandre Vermeerbergen
>>
>

-- 
*  Regards*
*  Sandeep Nemuri*

Mime
View raw message