atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ayub Khan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ATLAS-675) Storm Hook should use timetsamps as Date type instead of Long
Date Thu, 18 Aug 2016 10:00:28 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426197#comment-15426197
] 

Ayub Khan commented on ATLAS-675:
---------------------------------

Verified the storm topology submission, everything is fine and UI also looks good. No changes
needed on UI.

{noformat}
[root@atlas-resilience-testing-1 ~]# storm jar storm-samples-1.0-jar-with-dependencies.jar
com.dsinpractice.storm.samples.WordCountTopology --cluster true --name erie_demo_topology
--path /user/storm/storm-hdfs-test --topic my-topic
Running: /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -server -Ddaemon.name= -Dstorm.options=
-Dstorm.home=/grid/0/hdp/2.5.0.0-1206/storm -Dstorm.log.dir=/grid/0/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dstorm.conf.file= -cp /grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/zookeeper.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/disruptor-3.3.2.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/clojure-1.7.0.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/storm-core-1.0.1.2.5.0.0-1206.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-over-slf4j-1.6.6.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/asm-5.0.3.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/objenesis-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1206.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/minlog-1.3.0.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/kryo-3.0.3.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/reflectasm-1.10.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-core-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/ambari-metrics-storm-sink.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/ring-cors-0.1.5.jar:/grid/0/hdp/2.5.0.0-1206/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1206.jar:/grid/0/hdp/2.5.0.0-1206/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1206.jar
org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer
storm-samples-1.0-jar-with-dependencies.jar /tmp/c35a6fe4652211e68963fa163ecea884.jar
Running: /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -client -Ddaemon.name= -Dstorm.options=
-Dstorm.home=/grid/0/hdp/2.5.0.0-1206/storm -Dstorm.log.dir=/grid/0/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
-Dstorm.conf.file= -cp /grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/zookeeper.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/disruptor-3.3.2.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/clojure-1.7.0.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/storm-core-1.0.1.2.5.0.0-1206.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-over-slf4j-1.6.6.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/asm-5.0.3.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/objenesis-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1206.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/minlog-1.3.0.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/kryo-3.0.3.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/reflectasm-1.10.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-core-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/ambari-metrics-storm-sink.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/ring-cors-0.1.5.jar:/grid/0/hdp/2.5.0.0-1206/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1206.jar:/grid/0/hdp/2.5.0.0-1206/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1206.jar:/tmp/c35a6fe4652211e68963fa163ecea884.jar:/usr/hdp/current/storm-supervisor/conf:/grid/0/hdp/2.5.0.0-1206/storm/bin
-Dstorm.jar=/tmp/c35a6fe4652211e68963fa163ecea884.jar com.dsinpractice.storm.samples.WordCountTopology
--cluster true --name erie_demo_topology --path /user/storm/storm-hdfs-test --topic my-topic
1539 [main] INFO  o.a.s.u.TupleUtils - Enabling tick tuple with interval [15]
1756 [main] INFO  o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest:
-8642044523715052640:-8214783921473533571
1871 [main] INFO  o.a.s.s.a.AuthUtils - Got AutoCreds []
2033 [main] INFO  o.a.s.m.n.Login - successfully logged in.
30005 [main] INFO  o.a.s.m.n.Login - successfully logged in.
30054 [main] INFO  o.a.s.m.n.Login - successfully logged in.
30107 [main] INFO  o.a.s.m.n.Login - successfully logged in.
30375 [main] INFO  o.a.s.StormSubmitter - Uploading topology jar /tmp/c35a6fe4652211e68963fa163ecea884.jar
to assigned location: /hadoop/storm/nimbus/inbox/stormjar-6463b40a-e1b4-418a-bc6b-ba3dbf306a0a.jar
31268 [main] INFO  o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location:
/hadoop/storm/nimbus/inbox/stormjar-6463b40a-e1b4-418a-bc6b-ba3dbf306a0a.jar
31268 [main] INFO  o.a.s.StormSubmitter - Submitting topology erie_demo_topology in distributed
mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-8642044523715052640:-8214783921473533571","topology.workers":3,"topology.debug":true}
31282 [main] INFO  o.a.s.m.n.Login - successfully logged in.
31328 [main] INFO  o.a.s.m.n.Login - successfully logged in.
32186 [main] INFO  o.a.s.StormSubmitter - Finished submitting topology: erie_demo_topology
32186 [main] INFO  o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
32786 [main] INFO  o.a.a.ApplicationProperties - Looking for atlas-application.properties
in classpath
32786 [main] INFO  o.a.a.ApplicationProperties - Loading atlas-application.properties from
file:/etc/storm/2.5.0.0-1206/0/atlas-application.properties
log4j:WARN No appenders could be found for logger (org.apache.atlas.ApplicationProperties).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
34277 [main] INFO  o.a.a.h.AtlasHook - Created Atlas Hook
34306 [main] INFO  o.a.s.m.n.Login - successfully logged in.
34364 [main] INFO  o.a.s.m.n.Login - successfully logged in.
34560 [main] INFO  o.a.s.StormSubmitter - Invoking the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
34561 [main] INFO  o.a.a.s.h.StormAtlasHook - Collecting metadata for a new storm topology:
erie_demo_topology
35160 [main] INFO  o.a.a.h.AtlasHook - Adding entity for type: kafka_topic
36470 [main] INFO  o.a.a.h.AtlasHook - Adding entity for type: hdfs_path
36477 [main] INFO  o.a.a.h.AtlasHook - Adding entity for type: storm_topology
36648 [main] INFO  o.a.k.c.p.ProducerConfig - ProducerConfig values:
       	interceptor.classes = null
       	request.timeout.ms = 30000
       	ssl.truststore.password = null
       	retry.backoff.ms = 100
       	buffer.memory = 33554432
       	batch.size = 16384
       	ssl.keymanager.algorithm = SunX509
       	receive.buffer.bytes = 32768
       	ssl.key.password = null
       	ssl.cipher.suites = null
       	sasl.kerberos.ticket.renew.jitter = 0.05
       	sasl.kerberos.service.name = kafka
       	ssl.provider = null
       	max.in.flight.requests.per.connection = 5
       	sasl.kerberos.ticket.renew.window.factor = 0.8
       	sasl.mechanism = GSSAPI
       	bootstrap.servers = [atlas-resilience-testing-1.openstacklocal:6667, atlas-resilience-testing-2.openstacklocal:6667]
       	client.id =
       	max.request.size = 1048576
       	acks = 1
       	linger.ms = 0
       	sasl.kerberos.kinit.cmd = /usr/bin/kinit
       	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
       	metadata.fetch.timeout.ms = 60000
       	ssl.endpoint.identification.algorithm = null
       	ssl.keystore.location = null
       	value.serializer = class org.apache.kafka.common.serialization.StringSerializer
       	ssl.truststore.location = null
       	ssl.keystore.password = null
       	block.on.buffer.full = false
       	key.serializer = class org.apache.kafka.common.serialization.StringSerializer
       	metrics.sample.window.ms = 30000
       	security.protocol = PLAINTEXTSASL
       	metadata.max.age.ms = 300000
       	ssl.protocol = TLS
       	sasl.kerberos.min.time.before.relogin = 60000
       	timeout.ms = 30000
       	connections.max.idle.ms = 540000
       	ssl.trustmanager.algorithm = PKIX
       	metric.reporters = []
       	ssl.truststore.type = JKS
       	compression.type = none
       	retries = 0
       	max.block.ms = 60000
       	partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
       	send.buffer.bytes = 131072
       	reconnect.backoff.ms = 50
       	metrics.num.samples = 2
       	ssl.keystore.type = JKS

36698 [main] INFO  o.a.k.c.s.a.AbstractLogin - Successfully logged in.
36701 [kafka-kerberos-refresh-thread] INFO  o.a.k.c.s.k.KerberosLogin - TGT refresh thread
started.
36703 [kafka-kerberos-refresh-thread] INFO  o.a.k.c.s.k.KerberosLogin - TGT valid starting
at: 2016-08-18T09:05:37.000+0000
36704 [kafka-kerberos-refresh-thread] INFO  o.a.k.c.s.k.KerberosLogin - TGT expires: 2016-08-18T19:05:37.000+0000
36704 [kafka-kerberos-refresh-thread] INFO  o.a.k.c.s.k.KerberosLogin - TGT refresh sleeping
until: 2016-08-18T17:19:01.968+0000
36727 [main] INFO  o.a.k.c.p.ProducerConfig - ProducerConfig values:
       	interceptor.classes = null
       	request.timeout.ms = 30000
       	ssl.truststore.password = null
       	retry.backoff.ms = 100
       	buffer.memory = 33554432
       	batch.size = 16384
       	ssl.keymanager.algorithm = SunX509
       	receive.buffer.bytes = 32768
       	ssl.key.password = null
       	ssl.cipher.suites = null
       	sasl.kerberos.ticket.renew.jitter = 0.05
       	sasl.kerberos.service.name = kafka
       	ssl.provider = null
       	max.in.flight.requests.per.connection = 5
       	sasl.kerberos.ticket.renew.window.factor = 0.8
       	sasl.mechanism = GSSAPI
       	bootstrap.servers = [atlas-resilience-testing-1.openstacklocal:6667, atlas-resilience-testing-2.openstacklocal:6667]
       	client.id = producer-1
       	max.request.size = 1048576
       	acks = 1
       	linger.ms = 0
       	sasl.kerberos.kinit.cmd = /usr/bin/kinit
       	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
       	metadata.fetch.timeout.ms = 60000
       	ssl.endpoint.identification.algorithm = null
       	ssl.keystore.location = null
       	value.serializer = class org.apache.kafka.common.serialization.StringSerializer
       	ssl.truststore.location = null
       	ssl.keystore.password = null
       	block.on.buffer.full = false
       	key.serializer = class org.apache.kafka.common.serialization.StringSerializer
       	metrics.sample.window.ms = 30000
       	security.protocol = PLAINTEXTSASL
       	metadata.max.age.ms = 300000
       	ssl.protocol = TLS
       	sasl.kerberos.min.time.before.relogin = 60000
       	timeout.ms = 30000
       	connections.max.idle.ms = 540000
       	ssl.trustmanager.algorithm = PKIX
       	metric.reporters = []
       	ssl.truststore.type = JKS
       	compression.type = none
       	retries = 0
       	max.block.ms = 60000
       	partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
       	send.buffer.bytes = 131072
       	reconnect.backoff.ms = 50
       	metrics.num.samples = 2
       	ssl.keystore.type = JKS

36731 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration zookeeper.sync.time.ms = 20
was supplied but isn't a known config.
36731 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration hook.group.id = atlas was
supplied but isn't a known config.
36732 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration zookeeper.connection.timeout.ms
= 200 was supplied but isn't a known config.
36732 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration zookeeper.session.timeout.ms
= 400 was supplied but isn't a known config.
36732 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration zookeeper.connect = atlas-resilience-testing-2.openstacklocal:2181,atlas-resilience-testing-1.openstacklocal:2181
was supplied but isn't a known config.
36732 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration key.deserializer = org.apache.kafka.common.serialization.StringDeserializer
was supplied but isn't a known config.
36733 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration auto.offset.reset = smallest
was supplied but isn't a known config.
36733 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration value.deserializer = org.apache.kafka.common.serialization.StringDeserializer
was supplied but isn't a known config.
36733 [main] WARN  o.a.k.c.p.ProducerConfig - The configuration partition.assignment.strategy
= roundrobin was supplied but isn't a known config.
36736 [main] INFO  o.a.k.c.u.AppInfoParser - Kafka version : 0.10.0.2.5.0.0-1206
36736 [main] INFO  o.a.k.c.u.AppInfoParser - Kafka commitId : 022ed507ec080025
37169 [kafka-producer-network-thread | producer-1] WARN  o.a.k.c.NetworkClient - Error while
fetching metadata with correlation id 0 : {ATLAS_HOOK=LEADER_NOT_AVAILABLE}
[root@atlas-resilience-testing-1 ~]#
{noformat}

Attaching UI snapshot as well. 

> Storm Hook should use timetsamps as Date type instead of Long
> -------------------------------------------------------------
>
>                 Key: ATLAS-675
>                 URL: https://issues.apache.org/jira/browse/ATLAS-675
>             Project: Atlas
>          Issue Type: Bug
>    Affects Versions: trunk, 0.7-incubating
>            Reporter: Suma Shivaprasad
>            Assignee: Ayub Khan
>             Fix For: trunk
>
>         Attachments: ATLAS-675-1.patch, ATLAS-675-2.patch, ATLAS-675-3.patch, ATLAS-675.patch,
Atlas 2016-08-18 15-24-55.png
>
>
> Storm topology startTime and endTime should be of type Date



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message