Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 33EFE200B66 for ; Thu, 18 Aug 2016 12:00:37 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 328F3160AAE; Thu, 18 Aug 2016 10:00:37 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2D2A6160AAB for ; Thu, 18 Aug 2016 12:00:36 +0200 (CEST) Received: (qmail 88551 invoked by uid 500); 18 Aug 2016 10:00:35 -0000 Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@atlas.incubator.apache.org Delivered-To: mailing list dev@atlas.incubator.apache.org Received: (qmail 88535 invoked by uid 99); 18 Aug 2016 10:00:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Aug 2016 10:00:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E968D1804CC for ; Thu, 18 Aug 2016 10:00:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.646 X-Spam-Level: X-Spam-Status: No, score=-4.646 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id IiEsen_0gyeo for ; Thu, 18 Aug 2016 10:00:30 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 0949C5F23D for ; Thu, 18 Aug 2016 10:00:28 +0000 (UTC) Received: (qmail 87938 invoked by uid 99); 18 Aug 2016 10:00:28 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Aug 2016 10:00:28 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 0625D2C0032 for ; Thu, 18 Aug 2016 10:00:28 +0000 (UTC) Date: Thu, 18 Aug 2016 10:00:28 +0000 (UTC) From: "Ayub Khan (JIRA)" To: dev@atlas.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ATLAS-675) Storm Hook should use timetsamps as Date type instead of Long MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 18 Aug 2016 10:00:37 -0000 [ https://issues.apache.org/jira/browse/ATLAS-675?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15426= 197#comment-15426197 ]=20 Ayub Khan commented on ATLAS-675: --------------------------------- Verified the storm topology submission, everything is fine and UI also look= s good. No changes needed on UI. {noformat} [root@atlas-resilience-testing-1 ~]# storm jar storm-samples-1.0-jar-with-d= ependencies.jar com.dsinpractice.storm.samples.WordCountTopology --cluster = true --name erie_demo_topology --path /user/storm/storm-hdfs-test --topic m= y-topic Running: /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -server -Ddaemon.n= ame=3D -Dstorm.options=3D -Dstorm.home=3D/grid/0/hdp/2.5.0.0-1206/storm -Ds= torm.log.dir=3D/grid/0/log/storm -Djava.library.path=3D/usr/local/lib:/opt/= local/lib:/usr/lib -Dstorm.conf.file=3D -cp /grid/0/hdp/2.5.0.0-1206/storm/= lib/log4j-api-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/zookeeper.jar:/gri= d/0/hdp/2.5.0.0-1206/storm/lib/disruptor-3.3.2.jar:/grid/0/hdp/2.5.0.0-1206= /storm/lib/clojure-1.7.0.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/storm-core-= 1.0.1.2.5.0.0-1206.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-over-slf4j-= 1.6.6.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/asm-5.0.3.jar:/grid/0/hdp/2.5.= 0.0-1206/storm/lib/objenesis-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/sto= rm-rename-hack-1.0.1.2.5.0.0-1206.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/mi= nlog-1.3.0.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-slf4j-impl-2.1.jar:= /grid/0/hdp/2.5.0.0-1206/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.5.0.0-= 1206/storm/lib/kryo-3.0.3.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/reflectasm= -1.10.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-core-2.1.jar:/grid/0/h= dp/2.5.0.0-1206/storm/lib/ambari-metrics-storm-sink.jar:/grid/0/hdp/2.5.0.0= -1206/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/ring= -cors-0.1.5.jar:/grid/0/hdp/2.5.0.0-1206/storm/extlib/atlas-plugin-classloa= der-0.7.0.2.5.0.0-1206.jar:/grid/0/hdp/2.5.0.0-1206/storm/extlib/storm-brid= ge-shim-0.7.0.2.5.0.0-1206.jar org.apache.storm.daemon.ClientJarTransformer= Runner org.apache.storm.hack.StormShadeTransformer storm-samples-1.0-jar-wi= th-dependencies.jar /tmp/c35a6fe4652211e68963fa163ecea884.jar Running: /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -client -Ddaemon.n= ame=3D -Dstorm.options=3D -Dstorm.home=3D/grid/0/hdp/2.5.0.0-1206/storm -Ds= torm.log.dir=3D/grid/0/log/storm -Djava.library.path=3D/usr/local/lib:/opt/= local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file=3D -= cp /grid/0/hdp/2.5.0.0-1206/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.5.0.0= -1206/storm/lib/zookeeper.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/disruptor-= 3.3.2.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/clojure-1.7.0.jar:/grid/0/hdp/= 2.5.0.0-1206/storm/lib/storm-core-1.0.1.2.5.0.0-1206.jar:/grid/0/hdp/2.5.0.= 0-1206/storm/lib/log4j-over-slf4j-1.6.6.jar:/grid/0/hdp/2.5.0.0-1206/storm/= lib/asm-5.0.3.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/objenesis-2.1.jar:/gri= d/0/hdp/2.5.0.0-1206/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1206.jar:/gr= id/0/hdp/2.5.0.0-1206/storm/lib/minlog-1.3.0.jar:/grid/0/hdp/2.5.0.0-1206/s= torm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/servle= t-api-2.5.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/kryo-3.0.3.jar:/grid/0/hdp= /2.5.0.0-1206/storm/lib/reflectasm-1.10.1.jar:/grid/0/hdp/2.5.0.0-1206/stor= m/lib/log4j-core-2.1.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/ambari-metrics-= storm-sink.jar:/grid/0/hdp/2.5.0.0-1206/storm/lib/slf4j-api-1.7.7.jar:/grid= /0/hdp/2.5.0.0-1206/storm/lib/ring-cors-0.1.5.jar:/grid/0/hdp/2.5.0.0-1206/= storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1206.jar:/grid/0/hdp/2.= 5.0.0-1206/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1206.jar:/tmp/c35a6= fe4652211e68963fa163ecea884.jar:/usr/hdp/current/storm-supervisor/conf:/gri= d/0/hdp/2.5.0.0-1206/storm/bin -Dstorm.jar=3D/tmp/c35a6fe4652211e68963fa163= ecea884.jar com.dsinpractice.storm.samples.WordCountTopology --cluster true= --name erie_demo_topology --path /user/storm/storm-hdfs-test --topic my-to= pic 1539 [main] INFO o.a.s.u.TupleUtils - Enabling tick tuple with interval [1= 5] 1756 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload= for MD5-digest: -8642044523715052640:-8214783921473533571 1871 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds [] 2033 [main] INFO o.a.s.m.n.Login - successfully logged in. 30005 [main] INFO o.a.s.m.n.Login - successfully logged in. 30054 [main] INFO o.a.s.m.n.Login - successfully logged in. 30107 [main] INFO o.a.s.m.n.Login - successfully logged in. 30375 [main] INFO o.a.s.StormSubmitter - Uploading topology jar /tmp/c35a6= fe4652211e68963fa163ecea884.jar to assigned location: /hadoop/storm/nimbus/= inbox/stormjar-6463b40a-e1b4-418a-bc6b-ba3dbf306a0a.jar 31268 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology ja= r to assigned location: /hadoop/storm/nimbus/inbox/stormjar-6463b40a-e1b4-4= 18a-bc6b-ba3dbf306a0a.jar 31268 [main] INFO o.a.s.StormSubmitter - Submitting topology erie_demo_top= ology in distributed mode with conf {"storm.zookeeper.topology.auth.scheme"= :"digest","storm.zookeeper.topology.auth.payload":"-8642044523715052640:-82= 14783921473533571","topology.workers":3,"topology.debug":true} 31282 [main] INFO o.a.s.m.n.Login - successfully logged in. 31328 [main] INFO o.a.s.m.n.Login - successfully logged in. 32186 [main] INFO o.a.s.StormSubmitter - Finished submitting topology: eri= e_demo_topology 32186 [main] INFO o.a.s.StormSubmitter - Initializing the registered ISubm= itterHook [org.apache.atlas.storm.hook.StormAtlasHook] 32786 [main] INFO o.a.a.ApplicationProperties - Looking for atlas-applicat= ion.properties in classpath 32786 [main] INFO o.a.a.ApplicationProperties - Loading atlas-application.= properties from file:/etc/storm/2.5.0.0-1206/0/atlas-application.properties log4j:WARN No appenders could be found for logger (org.apache.atlas.Applica= tionProperties). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for mo= re info. 34277 [main] INFO o.a.a.h.AtlasHook - Created Atlas Hook 34306 [main] INFO o.a.s.m.n.Login - successfully logged in. 34364 [main] INFO o.a.s.m.n.Login - successfully logged in. 34560 [main] INFO o.a.s.StormSubmitter - Invoking the registered ISubmitte= rHook [org.apache.atlas.storm.hook.StormAtlasHook] 34561 [main] INFO o.a.a.s.h.StormAtlasHook - Collecting metadata for a new= storm topology: erie_demo_topology 35160 [main] INFO o.a.a.h.AtlasHook - Adding entity for type: kafka_topic 36470 [main] INFO o.a.a.h.AtlasHook - Adding entity for type: hdfs_path 36477 [main] INFO o.a.a.h.AtlasHook - Adding entity for type: storm_topolo= gy 36648 [main] INFO o.a.k.c.p.ProducerConfig - ProducerConfig values: =09interceptor.classes =3D null =09request.timeout.ms =3D 30000 =09ssl.truststore.password =3D null =09retry.backoff.ms =3D 100 =09buffer.memory =3D 33554432 =09batch.size =3D 16384 =09ssl.keymanager.algorithm =3D SunX509 =09receive.buffer.bytes =3D 32768 =09ssl.key.password =3D null =09ssl.cipher.suites =3D null =09sasl.kerberos.ticket.renew.jitter =3D 0.05 =09sasl.kerberos.service.name =3D kafka =09ssl.provider =3D null =09max.in.flight.requests.per.connection =3D 5 =09sasl.kerberos.ticket.renew.window.factor =3D 0.8 =09sasl.mechanism =3D GSSAPI =09bootstrap.servers =3D [atlas-resilience-testing-1.openstacklocal:= 6667, atlas-resilience-testing-2.openstacklocal:6667] =09client.id =3D =09max.request.size =3D 1048576 =09acks =3D 1 =09linger.ms =3D 0 =09sasl.kerberos.kinit.cmd =3D /usr/bin/kinit =09ssl.enabled.protocols =3D [TLSv1.2, TLSv1.1, TLSv1] =09metadata.fetch.timeout.ms =3D 60000 =09ssl.endpoint.identification.algorithm =3D null =09ssl.keystore.location =3D null =09value.serializer =3D class org.apache.kafka.common.serialization.= StringSerializer =09ssl.truststore.location =3D null =09ssl.keystore.password =3D null =09block.on.buffer.full =3D false =09key.serializer =3D class org.apache.kafka.common.serialization.St= ringSerializer =09metrics.sample.window.ms =3D 30000 =09security.protocol =3D PLAINTEXTSASL =09metadata.max.age.ms =3D 300000 =09ssl.protocol =3D TLS =09sasl.kerberos.min.time.before.relogin =3D 60000 =09timeout.ms =3D 30000 =09connections.max.idle.ms =3D 540000 =09ssl.trustmanager.algorithm =3D PKIX =09metric.reporters =3D [] =09ssl.truststore.type =3D JKS =09compression.type =3D none =09retries =3D 0 =09max.block.ms =3D 60000 =09partitioner.class =3D class org.apache.kafka.clients.producer.int= ernals.DefaultPartitioner =09send.buffer.bytes =3D 131072 =09reconnect.backoff.ms =3D 50 =09metrics.num.samples =3D 2 =09ssl.keystore.type =3D JKS 36698 [main] INFO o.a.k.c.s.a.AbstractLogin - Successfully logged in. 36701 [kafka-kerberos-refresh-thread] INFO o.a.k.c.s.k.KerberosLogin - TGT= refresh thread started. 36703 [kafka-kerberos-refresh-thread] INFO o.a.k.c.s.k.KerberosLogin - TGT= valid starting at: 2016-08-18T09:05:37.000+0000 36704 [kafka-kerberos-refresh-thread] INFO o.a.k.c.s.k.KerberosLogin - TGT= expires: 2016-08-18T19:05:37.000+0000 36704 [kafka-kerberos-refresh-thread] INFO o.a.k.c.s.k.KerberosLogin - TGT= refresh sleeping until: 2016-08-18T17:19:01.968+0000 36727 [main] INFO o.a.k.c.p.ProducerConfig - ProducerConfig values: =09interceptor.classes =3D null =09request.timeout.ms =3D 30000 =09ssl.truststore.password =3D null =09retry.backoff.ms =3D 100 =09buffer.memory =3D 33554432 =09batch.size =3D 16384 =09ssl.keymanager.algorithm =3D SunX509 =09receive.buffer.bytes =3D 32768 =09ssl.key.password =3D null =09ssl.cipher.suites =3D null =09sasl.kerberos.ticket.renew.jitter =3D 0.05 =09sasl.kerberos.service.name =3D kafka =09ssl.provider =3D null =09max.in.flight.requests.per.connection =3D 5 =09sasl.kerberos.ticket.renew.window.factor =3D 0.8 =09sasl.mechanism =3D GSSAPI =09bootstrap.servers =3D [atlas-resilience-testing-1.openstacklocal:= 6667, atlas-resilience-testing-2.openstacklocal:6667] =09client.id =3D producer-1 =09max.request.size =3D 1048576 =09acks =3D 1 =09linger.ms =3D 0 =09sasl.kerberos.kinit.cmd =3D /usr/bin/kinit =09ssl.enabled.protocols =3D [TLSv1.2, TLSv1.1, TLSv1] =09metadata.fetch.timeout.ms =3D 60000 =09ssl.endpoint.identification.algorithm =3D null =09ssl.keystore.location =3D null =09value.serializer =3D class org.apache.kafka.common.serialization.= StringSerializer =09ssl.truststore.location =3D null =09ssl.keystore.password =3D null =09block.on.buffer.full =3D false =09key.serializer =3D class org.apache.kafka.common.serialization.St= ringSerializer =09metrics.sample.window.ms =3D 30000 =09security.protocol =3D PLAINTEXTSASL =09metadata.max.age.ms =3D 300000 =09ssl.protocol =3D TLS =09sasl.kerberos.min.time.before.relogin =3D 60000 =09timeout.ms =3D 30000 =09connections.max.idle.ms =3D 540000 =09ssl.trustmanager.algorithm =3D PKIX =09metric.reporters =3D [] =09ssl.truststore.type =3D JKS =09compression.type =3D none =09retries =3D 0 =09max.block.ms =3D 60000 =09partitioner.class =3D class org.apache.kafka.clients.producer.int= ernals.DefaultPartitioner =09send.buffer.bytes =3D 131072 =09reconnect.backoff.ms =3D 50 =09metrics.num.samples =3D 2 =09ssl.keystore.type =3D JKS 36731 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.s= ync.time.ms =3D 20 was supplied but isn't a known config. 36731 [main] WARN o.a.k.c.p.ProducerConfig - The configuration hook.group.= id =3D atlas was supplied but isn't a known config. 36732 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.c= onnection.timeout.ms =3D 200 was supplied but isn't a known config. 36732 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.s= ession.timeout.ms =3D 400 was supplied but isn't a known config. 36732 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.c= onnect =3D atlas-resilience-testing-2.openstacklocal:2181,atlas-resilience-= testing-1.openstacklocal:2181 was supplied but isn't a known config. 36732 [main] WARN o.a.k.c.p.ProducerConfig - The configuration key.deseria= lizer =3D org.apache.kafka.common.serialization.StringDeserializer was supp= lied but isn't a known config. 36733 [main] WARN o.a.k.c.p.ProducerConfig - The configuration auto.offset= .reset =3D smallest was supplied but isn't a known config. 36733 [main] WARN o.a.k.c.p.ProducerConfig - The configuration value.deser= ializer =3D org.apache.kafka.common.serialization.StringDeserializer was su= pplied but isn't a known config. 36733 [main] WARN o.a.k.c.p.ProducerConfig - The configuration partition.a= ssignment.strategy =3D roundrobin was supplied but isn't a known config. 36736 [main] INFO o.a.k.c.u.AppInfoParser - Kafka version : 0.10.0.2.5.0.0= -1206 36736 [main] INFO o.a.k.c.u.AppInfoParser - Kafka commitId : 022ed507ec080= 025 37169 [kafka-producer-network-thread | producer-1] WARN o.a.k.c.NetworkCli= ent - Error while fetching metadata with correlation id 0 : {ATLAS_HOOK=3DL= EADER_NOT_AVAILABLE} [root@atlas-resilience-testing-1 ~]# {noformat} Attaching UI snapshot as well.=20 > Storm Hook should use timetsamps as Date type instead of Long > ------------------------------------------------------------- > > Key: ATLAS-675 > URL: https://issues.apache.org/jira/browse/ATLAS-675 > Project: Atlas > Issue Type: Bug > Affects Versions: trunk, 0.7-incubating > Reporter: Suma Shivaprasad > Assignee: Ayub Khan > Fix For: trunk > > Attachments: ATLAS-675-1.patch, ATLAS-675-2.patch, ATLAS-675-3.pa= tch, ATLAS-675.patch, Atlas 2016-08-18 15-24-55.png > > > Storm topology startTime and endTime should be of type Date -- This message was sent by Atlassian JIRA (v6.3.4#6332)