atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ayub Khan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ATLAS-1121) NPE while submitting topology in StormHook
Date Tue, 16 Aug 2016 14:41:20 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422817#comment-15422817
] 

Ayub Khan commented on ATLAS-1121:
----------------------------------

Submitting the patch to handle NPE with respect to jackson-databind. [~madhan.neethiraj][~shwethags]
[~suma.shivaprasad] please review the patch

I have tested the patch and the topology submission is successful and the entities are created
over Atlas. You will still see the error but the topology metadata submission to Atlas succeeds..

{noformat}
[root@vimal-erie-6-1 storm-topology]# storm jar hcube-storm-topology-0.0.1.jar org.hw.hcube.storm.LogStreamingToplogyV2
phoenix.properties
Running: /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/grid/0/hdp/2.5.0.0-1181/storm
-Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dstorm.conf.file= -cp /grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-over-slf4j-1.6.6.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/zookeeper.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/reflectasm-1.10.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ambari-metrics-storm-sink.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/objenesis-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/clojure-1.7.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/disruptor-3.3.2.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-core-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/minlog-1.3.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/kryo-3.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/asm-5.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ring-cors-0.1.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-core-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1181.jar
org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer
hcube-storm-topology-0.0.1.jar /tmp/6a601f5663ba11e698cffa163ef6032d.jar



Running: /usr/jdk64/jdk1.8.0_60/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/grid/0/hdp/2.5.0.0-1181/storm
-Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
-Dstorm.conf.file= -cp /grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-over-slf4j-1.6.6.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/zookeeper.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/reflectasm-1.10.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ambari-metrics-storm-sink.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/objenesis-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/clojure-1.7.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/disruptor-3.3.2.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-core-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/minlog-1.3.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/kryo-3.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/asm-5.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ring-cors-0.1.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-core-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1181.jar:/tmp/6a601f5663ba11e698cffa163ef6032d.jar:/usr/hdp/current/storm-supervisor/conf:/grid/0/hdp/2.5.0.0-1181/storm/bin
-Dstorm.jar=/tmp/6a601f5663ba11e698cffa163ef6032d.jar org.hw.hcube.storm.LogStreamingToplogyV2
phoenix.properties
2339 [main] INFO  o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest:
-7910590019420719599:-5664763422779979058
2456 [main] INFO  o.a.s.s.a.AuthUtils - Got AutoCreds []
2543 [main] INFO  o.a.s.StormSubmitter - Uploading topology jar /tmp/6a601f5663ba11e698cffa163ef6032d.jar
to assigned location: /grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar
Start uploading file '/tmp/6a601f5663ba11e698cffa163ef6032d.jar' to '/grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar'
(161473843 bytes)
[==================================================] 161473843 / 161473843
File '/tmp/6a601f5663ba11e698cffa163ef6032d.jar' uploaded to '/grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar'
(161473843 bytes)
4418 [main] INFO  o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location:
/grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar
4419 [main] INFO  o.a.s.StormSubmitter - Submitting topology hcube-log-streaming-phoenix in
distributed mode with conf {"topology.workers":10,"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-7910590019420719599:-5664763422779979058"}
8342 [main] INFO  o.a.s.StormSubmitter - Finished submitting topology: hcube-log-streaming-phoenix
8349 [main] INFO  o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
8924 [main] INFO  o.a.a.ApplicationProperties - Looking for atlas-application.properties in
classpath
8924 [main] INFO  o.a.a.ApplicationProperties - Loading atlas-application.properties from
file:/etc/storm/2.5.0.0-1181/0/atlas-application.properties
log4j:WARN No appenders could be found for logger (org.apache.atlas.ApplicationProperties).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
10171 [main] INFO  o.a.a.h.AtlasHook - Created Atlas Hook
10522 [main] INFO  o.a.s.StormSubmitter - Invoking the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
10523 [main] INFO  o.a.a.s.h.StormAtlasHook - Collecting metadata for a new storm topology:
hcube-log-streaming-phoenix
10634 [main] ERROR o.a.a.s.h.StormAtlasHook - storm topology exception
java.lang.NullPointerException
       	at java.text.DateFormat.hashCode(DateFormat.java:739) ~[?:1.8.0_60]
       	at java.util.HashMap.hash(HashMap.java:338) ~[?:1.8.0_60]
       	at java.util.HashMap.put(HashMap.java:611) ~[?:1.8.0_60]
       	at java.util.HashSet.add(HashSet.java:219) ~[?:1.8.0_60]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:141)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormAtlasHook.createDataSet(StormAtlasHook.java:187)
[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyInputs(StormAtlasHook.java:149)
[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyDataSets(StormAtlasHook.java:132)
[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:89) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:57) [storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:0.7.0.2.5.0.0-1181]
       	at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:285) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
       	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:258) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
       	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
       	at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:347)
[storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
       	at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:328)
[storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
       	at org.hw.hcube.storm.LogStreamingToplogyV2.main(LogStreamingToplogyV2.java:164) [6a601f5663ba11e698cffa163ef6032d.jar:?]
10709 [main] ERROR o.a.a.s.h.StormAtlasHook - storm topology exception
java.lang.NullPointerException
       	at java.text.DateFormat.hashCode(DateFormat.java:739) ~[?:1.8.0_60]
       	at java.util.HashMap.hash(HashMap.java:338) ~[?:1.8.0_60]
       	at java.util.HashMap.put(HashMap.java:611) ~[?:1.8.0_60]
       	at java.util.HashSet.add(HashSet.java:219) ~[?:1.8.0_60]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:141)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198)
~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormAtlasHook.createSpoutInstance(StormAtlasHook.java:310)
[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormAtlasHook.addSpouts(StormAtlasHook.java:294) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormAtlasHook.createTopologyGraph(StormAtlasHook.java:280)
[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:95) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
       	at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:57) [storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:0.7.0.2.5.0.0-1181]
       	at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:285) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
       	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:258) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
       	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
       	at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:347)
[storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
       	at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:328)
[storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
       	at org.hw.hcube.storm.LogStreamingToplogyV2.main(LogStreamingToplogyV2.java:164) [6a601f5663ba11e698cffa163ef6032d.jar:?]
10718 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: org.apache.storm.jdbc.bolt.JdbcInsertBolt@7a5b769b
10718 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: org.apache.storm.jdbc.mapper.SimpleJdbcMapper@f4c0e4e
10718 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='host',
val=null, sqlType=12}
10718 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='type',
val=null, sqlType=12}
10718 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='seq_num',
val=null, sqlType=-5}
10718 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='id',
val=null, sqlType=12}
10719 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='cluster',
val=null, sqlType=12}
10719 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='event_count',
val=null, sqlType=-5}
10719 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='message_md5',
val=null, sqlType=12}
10719 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='event_md5',
val=null, sqlType=12}
10723 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='ip',
val=null, sqlType=12}
10725 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='path',
val=null, sqlType=12}
10725 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='file',
val=null, sqlType=12}
10725 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='method',
val=null, sqlType=12}
10725 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='logger_name',
val=null, sqlType=12}
10725 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='logtime',
val=null, sqlType=93}
10725 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='line_number',
val=null, sqlType=-5}
10732 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='logfile_line_number',
val=null, sqlType=-5}
10732 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='level',
val=null, sqlType=12}
10732 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='log_message',
val=null, sqlType=12}
10732 [main] INFO  o.a.a.s.h.StormTopologyUtil - Processing instance: org.apache.storm.jdbc.common.HikariCPConnectionProvider@24361cfc
10737 [main] INFO  o.a.a.h.AtlasHook - Adding entity for type: storm_topology
23946 [main] INFO  o.a.k.c.p.ProducerConfig - ProducerConfig values:
       	metric.reporters = []
       	metadata.max.age.ms = 300000
       	reconnect.backoff.ms = 50
       	sasl.kerberos.ticket.renew.window.factor = 0.8
       	bootstrap.servers = [vimal-erie-6-1.openstacklocal:6667]
       	ssl.keystore.type = JKS
       	sasl.mechanism = GSSAPI
       	max.block.ms = 60000
       	interceptor.classes = null
       	ssl.truststore.password = null
       	client.id =
       	ssl.endpoint.identification.algorithm = null
       	request.timeout.ms = 30000
       	acks = 1
       	receive.buffer.bytes = 32768
       	ssl.truststore.type = JKS
       	retries = 0
       	ssl.truststore.location = null
       	ssl.keystore.password = null
       	send.buffer.bytes = 131072
       	compression.type = none
       	metadata.fetch.timeout.ms = 60000
       	retry.backoff.ms = 100
       	sasl.kerberos.kinit.cmd = /usr/bin/kinit
       	buffer.memory = 33554432
       	timeout.ms = 30000
       	key.serializer = class org.apache.kafka.common.serialization.StringSerializer
       	sasl.kerberos.service.name = null
       	sasl.kerberos.ticket.renew.jitter = 0.05
       	ssl.trustmanager.algorithm = PKIX
       	block.on.buffer.full = false
       	ssl.key.password = null
       	sasl.kerberos.min.time.before.relogin = 60000
       	connections.max.idle.ms = 540000
       	max.in.flight.requests.per.connection = 5
       	metrics.num.samples = 2
       	ssl.protocol = TLS
       	ssl.provider = null
       	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
       	batch.size = 16384
       	ssl.keystore.location = null
       	ssl.cipher.suites = null
       	security.protocol = PLAINTEXT
       	max.request.size = 1048576
       	value.serializer = class org.apache.kafka.common.serialization.StringSerializer
       	ssl.keymanager.algorithm = SunX509
       	metrics.sample.window.ms = 30000
       	partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
       	linger.ms = 0

24084 [main] INFO  o.a.k.c.p.ProducerConfig - ProducerConfig values:
       	metric.reporters = []
       	metadata.max.age.ms = 300000
       	reconnect.backoff.ms = 50
       	sasl.kerberos.ticket.renew.window.factor = 0.8
       	bootstrap.servers = [vimal-erie-6-1.openstacklocal:6667]
       	ssl.keystore.type = JKS
       	sasl.mechanism = GSSAPI
       	max.block.ms = 60000
       	interceptor.classes = null
       	ssl.truststore.password = null
       	client.id = producer-1
       	ssl.endpoint.identification.algorithm = null
       	request.timeout.ms = 30000
       	acks = 1
       	receive.buffer.bytes = 32768
       	ssl.truststore.type = JKS
       	retries = 0
       	ssl.truststore.location = null
       	ssl.keystore.password = null
       	send.buffer.bytes = 131072
       	compression.type = none
       	metadata.fetch.timeout.ms = 60000
       	retry.backoff.ms = 100
       	sasl.kerberos.kinit.cmd = /usr/bin/kinit
       	buffer.memory = 33554432
       	timeout.ms = 30000
       	key.serializer = class org.apache.kafka.common.serialization.StringSerializer
       	sasl.kerberos.service.name = null
       	sasl.kerberos.ticket.renew.jitter = 0.05
       	ssl.trustmanager.algorithm = PKIX
       	block.on.buffer.full = false
       	ssl.key.password = null
       	sasl.kerberos.min.time.before.relogin = 60000
       	connections.max.idle.ms = 540000
       	max.in.flight.requests.per.connection = 5
       	metrics.num.samples = 2
       	ssl.protocol = TLS
       	ssl.provider = null
       	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
       	batch.size = 16384
       	ssl.keystore.location = null
       	ssl.cipher.suites = null
       	security.protocol = PLAINTEXT
       	max.request.size = 1048576
       	value.serializer = class org.apache.kafka.common.serialization.StringSerializer
       	ssl.keymanager.algorithm = SunX509
       	metrics.sample.window.ms = 30000
       	partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
       	linger.ms = 0

24089 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration key.deserializer = org.apache.kafka.common.serialization.StringDeserializer
was supplied but isn't a known config.
24090 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration value.deserializer = org.apache.kafka.common.serialization.StringDeserializer
was supplied but isn't a known config.
24090 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration hook.group.id = atlas was
supplied but isn't a known config.
24090 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration partition.assignment.strategy
= roundrobin was supplied but isn't a known config.
24090 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration zookeeper.connection.timeout.ms
= 200 was supplied but isn't a known config.
24091 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration zookeeper.session.timeout.ms
= 400 was supplied but isn't a known config.
24091 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration zookeeper.connect = vimal-erie-6-1.openstacklocal:2181
was supplied but isn't a known config.
24091 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration zookeeper.sync.time.ms = 20
was supplied but isn't a known config.
24091 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration auto.offset.reset = smallest
was supplied but isn't a known config.
24093 [main] INFO  o.a.k.c.u.AppInfoParser - Kafka version : 0.10.0.2.5.0.0-1181
24093 [main] INFO  o.a.k.c.u.AppInfoParser - Kafka commitId : 022ed507ec080025
{noformat}

Attaching Atlas UI snapshot showing the entities created as part of storm topology submission.
 !Atlas 2016-08-16 19-48-59.png|thumbnail! 

> NPE while submitting topology in StormHook
> ------------------------------------------
>
>                 Key: ATLAS-1121
>                 URL: https://issues.apache.org/jira/browse/ATLAS-1121
>             Project: Atlas
>          Issue Type: Bug
>    Affects Versions: trunk
>            Reporter: Ayub Khan
>            Assignee: Ayub Khan
>
> NPE while submitting topology in StormHook
> {code}
> [storm@ctr-e25-1471039652053-0001-01-000009 erie]$ storm jar hcube-storm-topology-0.0.1.jar
org.hw.hcube.storm.LogStreamingToplogyV2 phoenix.properties 
> Running: /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.5.0.0-1201/storm
-Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dstorm.conf.file= -cp /usr/hdp/2.5.0.0-1201/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.0.0-1201/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.0.0-1201/storm/lib/storm-core-1.0.1.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.0.0-1201/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.0.0-1201/storm/lib/zookeeper.jar:/usr/hdp/2.5.0.0-1201/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.0.0-1201/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.0.0-1201/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.0.0-1201/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.0.0-1201/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.5.0.0-1201/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1201.jar
org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer
hcube-storm-topology-0.0.1.jar /tmp/9e206b70636a11e685530242ac1b1bc0.jar
> Running: /usr/jdk64/jdk1.8.0_60/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.5.0.0-1201/storm
-Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
-Dstorm.conf.file= -cp /usr/hdp/2.5.0.0-1201/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.0.0-1201/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.0.0-1201/storm/lib/storm-core-1.0.1.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.0.0-1201/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.0.0-1201/storm/lib/zookeeper.jar:/usr/hdp/2.5.0.0-1201/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.0.0-1201/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.0.0-1201/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.0.0-1201/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.0.0-1201/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.5.0.0-1201/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1201.jar:/tmp/9e206b70636a11e685530242ac1b1bc0.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.5.0.0-1201/storm/bin
-Dstorm.jar=/tmp/9e206b70636a11e685530242ac1b1bc0.jar org.hw.hcube.storm.LogStreamingToplogyV2
phoenix.properties
> 1269 [main] INFO  o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest:
-5580857394466738431:-5449170806113196196
> 1320 [main] INFO  o.a.s.s.a.AuthUtils - Got AutoCreds []
> 1533 [main] INFO  o.a.s.m.n.Login - successfully logged in.
> 1682 [main] INFO  o.a.s.m.n.Login - successfully logged in.
> 1710 [main] INFO  o.a.s.m.n.Login - successfully logged in.
> 1738 [main] INFO  o.a.s.m.n.Login - successfully logged in.
> 1765 [main] INFO  o.a.s.StormSubmitter - Uploading topology jar /tmp/9e206b70636a11e685530242ac1b1bc0.jar
to assigned location: /hadoop/storm/nimbus/inbox/stormjar-2d8edcfa-000b-41d3-a170-9982b45867c0.jar
> Start uploading file '/tmp/9e206b70636a11e685530242ac1b1bc0.jar' to '/hadoop/storm/nimbus/inbox/stormjar-2d8edcfa-000b-41d3-a170-9982b45867c0.jar'
(161437864 bytes)
> [==================================================] 161437864 / 161437864
> File '/tmp/9e206b70636a11e685530242ac1b1bc0.jar' uploaded to '/hadoop/storm/nimbus/inbox/stormjar-2d8edcfa-000b-41d3-a170-9982b45867c0.jar'
(161437864 bytes)
> 5154 [main] INFO  o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned
location: /hadoop/storm/nimbus/inbox/stormjar-2d8edcfa-000b-41d3-a170-9982b45867c0.jar
> 5155 [main] INFO  o.a.s.StormSubmitter - Submitting topology hcube-log-streaming-phoenix
in distributed mode with conf {"topology.workers":10,"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-5580857394466738431:-5449170806113196196"}
> 5164 [main] INFO  o.a.s.m.n.Login - successfully logged in.
> 5194 [main] INFO  o.a.s.m.n.Login - successfully logged in.
> 6269 [main] INFO  o.a.s.StormSubmitter - Finished submitting topology: hcube-log-streaming-phoenix
> 6274 [main] INFO  o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
> 6366 [main] INFO  o.a.a.ApplicationProperties - Looking for atlas-application.properties
in classpath
> 6366 [main] INFO  o.a.a.ApplicationProperties - Loading atlas-application.properties
from file:/etc/storm/2.5.0.0-1201/0/atlas-application.properties
> log4j:WARN No appenders could be found for logger (org.apache.atlas.ApplicationProperties).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> 6736 [main] INFO  o.a.a.h.AtlasHook - Created Atlas Hook
> 6741 [main] INFO  o.a.s.m.n.Login - successfully logged in.
> 6771 [main] INFO  o.a.s.m.n.Login - successfully logged in.
> 6828 [main] INFO  o.a.s.StormSubmitter - Invoking the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
> 6829 [main] INFO  o.a.a.s.h.StormAtlasHook - Collecting metadata for a new storm topology:
hcube-log-streaming-phoenix
> 6859 [main] WARN  o.a.s.StormSubmitter - Error occurred in invoking submitter hook:[org.apache.atlas.storm.hook.StormAtlasHook]

> java.lang.RuntimeException: Atlas hook is unable to process the topology.
> 	at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:105) ~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> 	at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:57) ~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> 	at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:285) [storm-core-1.0.1.2.5.0.0-1201.jar:1.0.1.2.5.0.0-1201]
> 	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:258) [storm-core-1.0.1.2.5.0.0-1201.jar:1.0.1.2.5.0.0-1201]
> 	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311) [storm-core-1.0.1.2.5.0.0-1201.jar:1.0.1.2.5.0.0-1201]
> 	at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:347)
[storm-core-1.0.1.2.5.0.0-1201.jar:1.0.1.2.5.0.0-1201]
> 	at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:328)
[storm-core-1.0.1.2.5.0.0-1201.jar:1.0.1.2.5.0.0-1201]
> 	at org.hw.hcube.storm.LogStreamingToplogyV2.main(LogStreamingToplogyV2.java:164) [9e206b70636a11e685530242ac1b1bc0.jar:?]
> Caused by: java.lang.NullPointerException
> 	at java.text.DateFormat.hashCode(DateFormat.java:739) ~[?:1.8.0_60]
> 	at java.util.HashMap.hash(HashMap.java:338) ~[?:1.8.0_60]
> 	at java.util.HashMap.put(HashMap.java:611) ~[?:1.8.0_60]
> 	at java.util.HashSet.add(HashSet.java:219) ~[?:1.8.0_60]
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:139)
~[?:?]
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
~[?:?]
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
~[?:?]
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
~[?:?]
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
~[?:?]
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
~[?:?]
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
~[?:?]
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
~[?:?]
> 	at org.apache.atlas.storm.hook.StormAtlasHook.createDataSet(StormAtlasHook.java:182)
~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> 	at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyInputs(StormAtlasHook.java:149)
~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> 	at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyDataSets(StormAtlasHook.java:132)
~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> 	at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:89) ~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> 	... 7 more
> Exception in thread "main" org.apache.storm.hooks.SubmitterHookException: java.lang.RuntimeException:
Atlas hook is unable to process the topology.
> 	at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:289)
> 	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:258)
> 	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311)
> 	at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:347)
> 	at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:328)
> 	at org.hw.hcube.storm.LogStreamingToplogyV2.main(LogStreamingToplogyV2.java:164)
> Caused by: java.lang.RuntimeException: Atlas hook is unable to process the topology.
> 	at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:105)
> 	at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:57)
> 	at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:285)
> 	... 5 more
> Caused by: java.lang.NullPointerException
> 	at java.text.DateFormat.hashCode(DateFormat.java:739)
> 	at java.util.HashMap.hash(HashMap.java:338)
> 	at java.util.HashMap.put(HashMap.java:611)
> 	at java.util.HashSet.add(HashSet.java:219)
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:139)
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> 	at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> 	at org.apache.atlas.storm.hook.StormAtlasHook.createDataSet(StormAtlasHook.java:182)
> 	at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyInputs(StormAtlasHook.java:149)
> 	at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyDataSets(StormAtlasHook.java:132)
> 	at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:89)
> 	... 7 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message