storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raja.Aravapalli <Raja.Aravapa...@target.com>
Subject hdfsbolt in secure cluster failing with
Date Tue, 16 Feb 2016 07:37:36 GMT

Hi,

I have a hdfs bolt in my storm topology, it is running perfectly fine for some time, but suddently
it is failing with SASL security exceptions!!

I have followed the details outlined [Working with Secure HDFS - Using key tabs on all worker
hosts] at link http://storm.apache.org/documentation/storm-hdfs.html as we are on Kerberos
secured cluster.

Shared below is the code i am using to configure hdfs bolt !


builder.setBolt("hdfs-bolt-1", HDFSBolt, 1)
        .setNumTasks(30)
        .shuffleGrouping("prev-bolt-1");


builder.setBolt("hdfs-bolt-2", invalidRecsHDFSBolt, 1)
        .setNumTasks(4)
        .shuffleGrouping("prev-bolt-2");


Below is the exception:


java.io.IOException: IOException flush:java.io.IOException: Failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials
provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is:
"xxxxxxxxxxxx"; destination host is: "xxxxxxxx":8020;

        at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:2082) ~[stormjar.jar:na]

        at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1969) ~[stormjar.jar:na]

        at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.hsync(HdfsDataOutputStream.java:95)
~[stormjar.jar:na]

        at org.apache.storm.hdfs.bolt.HdfsBolt.execute(HdfsBolt.java:100) ~[stormjar.jar:na]

        at backtype.storm.daemon.executor$fn__5265$tuple_action_fn__5267.invoke(executor.clj:659)
[storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1]

        at backtype.storm.daemon.executor$mk_task_receiver$fn__5188.invoke(executor.clj:415)
[storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1]

        at backtype.storm.disruptor$clojure_handler$reify__1064.onEvent(disruptor.clj:58)
[storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1]

        at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
[storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1]

        at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
[storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1]

        at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
[storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1]

        at backtype.storm.daemon.executor$fn__5265$fn__5278$fn__5329.invoke(executor.clj:794)
[storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1]

        at backtype.storm.util$async_loop$fn__551.invoke(util.clj:465) [storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1]

        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]

        at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]


Some where i read, storm hdfs bolt fails to authenticate, when multiple tasks/executors used!!

Can someone pls help me fix the issue... Please share your thoughts.... to fix the issue.



Regards,
Raja Aravapalli

Mime
View raw message