Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 86565200B76 for ; Tue, 30 Aug 2016 14:36:32 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 84CA6160ABA; Tue, 30 Aug 2016 12:36:32 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0C78A160AA8 for ; Tue, 30 Aug 2016 14:36:30 +0200 (CEST) Received: (qmail 71662 invoked by uid 500); 30 Aug 2016 12:36:30 -0000 Mailing-List: contact dev-help@gearpump.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@gearpump.incubator.apache.org Delivered-To: mailing list dev@gearpump.incubator.apache.org Received: (qmail 71651 invoked by uid 99); 30 Aug 2016 12:36:30 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Aug 2016 12:36:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id CA3D9C05B7 for ; Tue, 30 Aug 2016 12:36:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.646 X-Spam-Level: X-Spam-Status: No, score=-4.646 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 2tm49C2TdUkw for ; Tue, 30 Aug 2016 12:36:24 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 09B6060D24 for ; Tue, 30 Aug 2016 12:36:22 +0000 (UTC) Received: (qmail 70981 invoked by uid 99); 30 Aug 2016 12:36:22 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Aug 2016 12:36:22 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 1614E2C1B80 for ; Tue, 30 Aug 2016 12:36:22 +0000 (UTC) Date: Tue, 30 Aug 2016 12:36:22 +0000 (UTC) From: "Kam Kasravi (JIRA)" To: dev@gearpump.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 30 Aug 2016 12:36:32 -0000 [ https://issues.apache.org/jira/browse/GEARPUMP-185?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15= 448900#comment-15448900 ]=20 Kam Kasravi commented on GEARPUMP-185: -------------------------------------- i'm able to confirm that the missing vmargs is related to the user argument= . Under the applications section of the dashboard UI - the vcap users appli= cation's config had no vmargs. Launching a new application (which ran as ya= rn) had a config that contained a valid vmargs for the executor > Yarn kills Apache Gearpump's worker if the memory usage is too high > ------------------------------------------------------------------- > > Key: GEARPUMP-185 > URL: https://issues.apache.org/jira/browse/GEARPUMP-185 > Project: Apache Gearpump > Issue Type: Bug > Environment: yarn > Reporter: Piotr Grabuszynski > Assignee: Kam Kasravi > Fix For: 0.8.0 > > > h3. Description > If the traffic is to big (memory usage is height) Yarn kills Gearpump's w= orker. > Clean YARN log (only the part with processes, bigger piece below): > {code} > Dump of the process-tree for container_e05_1468841606343_0128_01_000003 : > =09|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(= MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE > =09|- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840 /usr/java= /jdk1.8.0_72/jre/bin/java -Dgearpump.username=3Dvcap -Dgearpump.application= Id=3D1 -Dgearpump.executorId=3D0 -Dgearpump.master.starttime=3D2016-07-20-0= 8-36 -Dgearpump.log.application.dir=3Dlogs -Djava.net.preferIPv4Stack=3Dtru= e -Dgearpump.config.file=3D/tmp/gearpump6797336735445779094.conf ... > =09|- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626 /usr/java/jdk1= .8.0_72/jre/bin/java -Dgearpump.username=3Dvcap -Dgearpump.applicationId=3D= 1 -Dgearpump.executorId=3D-1 -Dgearpump.master.starttime=3D2016-07-20-08-36= -Dgearpump.log.application.dir=3Dlogs -Djava.net.preferIPv4Stack=3Dtrue -D= gearpump.config.file=3D/tmp/gearpump941454978115257558.conf ... > =09|- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c /usr/java= /jdk1.8.0_72/bin/java -Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/conf:pack= /gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoo= p/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAG= ER ... > =09|- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a /var= /log/hadoop-yarn/container/application_1468841606343_0128/container_e05_146= 8841606343_0128_01_000003/stderr=20 > =09|- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249 /usr/java/jdk1= .8.0_72/bin/java -Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/conf:pack/gearp= ump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf= .cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER ... > =09|- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356 /usr/java/jd= k1.8.0_72/jre/bin/java -Dgearpump.username=3Dvcap -Dgearpump.applicationId= =3D2 -Dgearpump.executorId=3D0 -Dgearpump.master.starttime=3D2016-07-20-08-= 36 -Dgearpump.log.application.dir=3Dlogs -Djava.net.preferIPv4Stack=3Dtrue = ... > =09|- 6360 2164 2159 2159 (java) 966 58 5845979136 54720 /usr/java/jdk1.8= .0_72/jre/bin/java -Dgearpump.username=3Dvcap -Dgearpump.applicationId=3D2 = -Dgearpump.executorId=3D-1 -Dgearpump.master.starttime=3D2016-07-20-08-36 -= Dgearpump.log.application.dir=3Dlogs -Djava.net.preferIPv4Stack=3Dtrue ... > {code} > *As we can see there are no "-Xmx" or "-Xsx" parameters near executors' p= rocesses.* > In standalone mode, we can see both: > {code} > // MASTER > GER\pgr+ 48280 5.3 3.1 5406960 261356 pts/13 Sl+ 09:28 0:10 /usr/lib= /jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=3Dtrue -Dlog= Filename=3Dmaster -Dgearpump.home=3D/home/local/GER/pgrabusz/Documents/gear= pump-sources/gearpump-2.11-0.8.0 -Djava.rmi.server.hostname=3Dlocalhost ... > // WORKER > GER\pgr+ 48417 7.7 1.8 5392212 154728 pts/27 Sl+ 09:29 0:05 /usr/lib= /jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=3Dtrue -Dlog= Filename=3Dworker -Dgearpump.home=3D/home/local/GER/pgrabusz/Documents/gear= pump-sources/gearpump-2.11-0.8.0 -Djava.rmi.server.hostname=3Dlocalhost ... > // APP MASTER > GER\pgr+ 48707 15.0 3.1 4572132 255776 pts/27 Sl+ 09:33 0:11 /usr/lib= /jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M -XX:+Heap= DumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFr= action=3D80 -XX:+UseParNewGC -XX:NewRatio=3D3 -Djava.rmi.server.hostname=3D= localhost -Dgearpump.username=3DGER\pgrabusz -Dgearpump.applicationId=3D1 -= Dgearpump.executorId=3D-1 ... > // APP EXECUTOR 0 > GER\pgr+ 48761 132 5.2 7268592 427200 pts/27 Sl+ 09:33 2:31 /usr/lib= /jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M -XX:+Heap= DumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFr= action=3D80 -XX:+UseParNewGC -XX:NewRatio=3D3 [...] -Dgearpump.applicationI= d=3D1 -Dgearpump.executorId=3D0 ... > {code} > h3. Steps to Reproduce > To generate trafic I've used such application: > {code} > kafka source > / \ > kafka sink hbase sink > {code} > Where kafka source write to the same topic as kafka source reads. > Send some message to topic. > Worker will disappear after few minutes (1-40 min depends on how much mem= ory worker has); > h3. Actual Results > Worker has been killed > h3. Additional information > Increasing value of env WORKERS_MEMORY_LIMIT in gearpump-broker doesn't w= ork. It increases only running time. > It might be a problem with some java properties (hip size or sth). It is = also possible there's some memory lick f.e. in hbase sink. > Bigger piece of YARN log: > {code} > 2016-07-20 09:04:22,217 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 7737 for container-id container_e05_1468841606343_0021_01_000002: 285.7 MB = of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used > 2016-07-20 09:04:22,270 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 046 for container-id container_e05_1468841606343_0128_01_000001: 356.7 MB o= f 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used > 2016-07-20 09:04:22,317 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 159 for container-id container_e05_1468841606343_0128_01_000003: 2.0 GB of = 2 GB physical memory used; 26.8 GB of 4.2 GB virtual memory used > 2016-07-20 09:04:22,378 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 116 for container-id container_e05_1468841606343_0128_01_000002: 348.0 MB o= f 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used > 2016-07-20 09:04:25,424 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 7737 for container-id container_e05_1468841606343_0021_01_000002: 285.7 MB = of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used > 2016-07-20 09:04:25,494 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 046 for container-id container_e05_1468841606343_0128_01_000001: 356.7 MB o= f 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used > 2016-07-20 09:04:25,538 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 159 for container-id container_e05_1468841606343_0128_01_000003: 2.0 GB of = 2 GB physical memory used; 26.8 GB of 4.2 GB virtual memory used > 2016-07-20 09:04:25,539 WARN org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Process tree for container: c= ontainer_e05_1468841606343_0128_01_000003 has processes older than 1 iterat= ion running over the configured limit. Limit=3D2147483648, current usage = =3D 2167832576 > 2016-07-20 09:04:25,547 WARN org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Container [pid=3D2159,contain= erID=3Dcontainer_e05_1468841606343_0128_01_000003] is running beyond physic= al memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 26.8 = GB of 4.2 GB virtual memory used. Killing container. > Dump of the process-tree for container_e05_1468841606343_0128_01_000003 : > =09|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(= MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE > =09|- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840 /usr/java= /jdk1.8.0_72/jre/bin/java -Dgearpump.username=3Dvcap -Dgearpump.application= Id=3D1 -Dgearpump.executorId=3D0 -Dgearpump.master.starttime=3D2016-07-20-0= 8-36 -Dgearpump.log.application.dir=3Dlogs -Djava.net.preferIPv4Stack=3Dtru= e -Dgearpump.config.file=3D/tmp/gearpump6797336735445779094.conf -cp conf:p= ack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final= .jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/ge= arpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0= .8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.= 0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1= .7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0= .8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parse= r-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2= .4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pac= k/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib= /akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.= 8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-= 0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.= 3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gear= pump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/l= ib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0= .7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpum= p-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.1= 1-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloud= era.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/clou= dera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.= 1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-f= ormat.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoo= p-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/had= oop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.= 11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5= .1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cd= h5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CD= H-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/c= loudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar= :/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift= .jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-ja= ckson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parqu= et-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/h= adoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/l= ib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p= 0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.= 1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5= .1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-= 1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CD= H-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/C= DH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera= /parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/clou= dera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/= opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotatio= ns-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/= hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.= 1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.= 1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/par= cels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/o= pt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/ > =09|- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626 /usr/java/jdk1= .8.0_72/jre/bin/java -Dgearpump.username=3Dvcap -Dgearpump.applicationId=3D= 1 -Dgearpump.executorId=3D-1 -Dgearpump.master.starttime=3D2016-07-20-08-36= -Dgearpump.log.application.dir=3Dlogs -Djava.net.preferIPv4Stack=3Dtrue -D= gearpump.config.file=3D/tmp/gearpump941454978115257558.conf -cp conf:pack/g= earpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:= pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpum= p-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/= lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib= /scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.= jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/= lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-com= binators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.= jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gea= rpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka= -agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar= :pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0= /lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.j= ar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-= 2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/ak= ka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.ja= r:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.1= 1-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8= .0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.Y= ARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/= parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar= :/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format= .jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-com= mon-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/p= arquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/li= b/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0= .11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.= 1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5= .1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloude= ra/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt= /cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:= /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson= .jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-ha= doop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop= /parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/ha= doop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/= lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.= 11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0= .11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh= 5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5= .1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.= 5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parc= els/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/= parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/c= loudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.= 6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoo= p/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.= 11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.= 11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/= CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cl= oudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/ > =09|- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c /usr/java= /jdk1.8.0_72/bin/java -Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/conf:pack= /gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoo= p/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAG= ER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/*:/opt/cloud= era/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/*:/opt/cloudera/parce= ls/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5= .5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.5.1= -1.cdh5.5.1.p0.11/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.= 5.1.p0.11/lib/hadoop-yarn/lib/* -Dgearpump.cluster.masters.0=3Dcdh-worker-2= .node.envname.consul:54946 -Dgearpump.log.daemon.dir=3D/var/log/hadoop-yarn= /container/application_1468841606343_0128/container_e05_1468841606343_0128_= 01_000003 -Dgearpump.worker-resource-manager-container-id=3Dcontainer_e05_1= 468841606343_0128_01_000003 -Dgearpump.home=3D/data1/hadoop/yarn/nm/usercac= he/cf/appcache/application_1468841606343_0128/container_e05_1468841606343_0= 128_01_000003/pack/gearpump-2.11-0.8.0 -Dgearpump.log.application.dir=3D/va= r/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_14= 68841606343_0128_01_000003 -Dgearpump.hostname=3Dcdh-worker-2.node.envname.= consul io.gearpump.cluster.main.Worker 2>&1 | /usr/bin/tee -a /var/log/ha= doop-yarn/container/application_1468841606343_0128/container_e05_1468841606= 343_0128_01_000003/stderr=20 > =09|- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a /var= /log/hadoop-yarn/container/application_1468841606343_0128/container_e05_146= 8841606343_0128_01_000003/stderr=20 > =09|- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249 /usr/java/jdk1= .8.0_72/bin/java -Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/conf:pack/gearp= ump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf= .cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/op= t/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/*:/opt/cloudera/pa= rcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH= -5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.5.1-1= .cdh5.5.1.p0.11/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh= 5.5.1.p0.11/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0= .11/lib/hadoop-yarn/lib/* -Dgearpump.cluster.masters.0=3Dcdh-worker-2.node.= envname.consul:54946 -Dgearpump.log.daemon.dir=3D/var/log/hadoop-yarn/conta= iner/application_1468841606343_0128/container_e05_1468841606343_0128_01_000= 003 -Dgearpump.worker-resource-manager-container-id=3Dcontainer_e05_1468841= 606343_0128_01_000003 -Dgearpump.home=3D/data1/hadoop/yarn/nm/usercache/cf/= appcache/application_1468841606343_0128/container_e05_1468841606343_0128_01= _000003/pack/gearpump-2.11-0.8.0 -Dgearpump.log.application.dir=3D/var/log/= hadoop-yarn/container/application_1468841606343_0128/container_e05_14688416= 06343_0128_01_000003 -Dgearpump.hostname=3Dcdh-worker-2.node.envname.consul= io.gearpump.cluster.main.Worker=20 > =09|- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356 /usr/java/jd= k1.8.0_72/jre/bin/java -Dgearpump.username=3Dvcap -Dgearpump.applicationId= =3D2 -Dgearpump.executorId=3D0 -Dgearpump.master.starttime=3D2016-07-20-08-= 36 -Dgearpump.log.application.dir=3Dlogs -Djava.net.preferIPv4Stack=3Dtrue = -Dgearpump.config.file=3D/tmp/gearpump7648852439078424047.conf -cp conf:pac= k/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.j= ar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gear= pump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8= .0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/= lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7= .7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8= .0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-= combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4= .2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/= gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/a= kka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.= jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.= 8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.= 0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpu= mp-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib= /akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7= .jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-= 2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-= 0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.clouder= a.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloude= ra/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.= jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-for= mat.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-= common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoo= p/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11= /lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1= .p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5= .5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-= 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/clo= udera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/= opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.j= ar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jack= son.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet= -hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/had= oop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib= /hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.= 11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.= p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1= .p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.= cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-= 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH= -5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/p= arcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloude= ra/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/op= t/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations= -2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/ha= doop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.= p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.= p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parce= ls/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt= /cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/ > =09|- 6360 2164 2159 2159 (java) 966 58 5845979136 54720 /usr/java/jdk1.8= .0_72/jre/bin/java -Dgearpump.username=3Dvcap -Dgearpump.applicationId=3D2 = -Dgearpump.executorId=3D-1 -Dgearpump.master.starttime=3D2016-07-20-08-36 -= Dgearpump.log.application.dir=3Dlogs -Djava.net.preferIPv4Stack=3Dtrue -Dge= arpump.config.file=3D/tmp/gearpump7311147404608349128.conf -cp conf:pack/ge= arpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:p= ack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump= -2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/l= ib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/= scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.j= ar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/l= ib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-comb= inators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.j= ar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gear= pump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-= agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:= pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/= lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.ja= r:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2= .11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akk= a-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar= :pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11= -0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.= 0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YA= RN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/p= arcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:= /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.= jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-comm= on-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/pa= rquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib= /hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.= 11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1= .p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.= 1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/clouder= a/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/= cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/= opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.= jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-had= oop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/= parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/had= oop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/l= ib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.1= 1/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.= 11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5= .5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.= 1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5= .1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parce= ls/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/p= arcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cl= oudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6= .0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop= /parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.1= 1/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.1= 1/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/C= DH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/clo= udera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib > 2016-07-20 09:04:25,548 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.container.Container: Container container_e05_1468841606343_0= 128_01_000003 transitioned from RUNNING to KILLING > 2016-07-20 09:04:25,548 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.launcher.ContainerLaunch: Cleaning up container container_e0= 5_1468841606343_0128_01_000003 > 2016-07-20 09:04:25,548 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Removed ProcessTree with root= 2159 > 2016-07-20 09:04:25,584 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 116 for container-id container_e05_1468841606343_0128_01_000002: 348.0 MB o= f 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used > 2016-07-20 09:04:25,621 WARN org.apache.hadoop.yarn.server.nodemanager.De= faultContainerExecutor: Exit code from container container_e05_146884160634= 3_0128_01_000003 is : 143 > 2016-07-20 09:04:25,655 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.container.Container: Container container_e05_1468841606343_0= 128_01_000003 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL > 2016-07-20 09:04:25,655 INFO org.apache.hadoop.yarn.server.nodemanager.De= faultContainerExecutor: Deleting absolute path : /data1/hadoop/yarn/nm/user= cache/cf/appcache/application_1468841606343_0128/container_e05_146884160634= 3_0128_01_000003 > 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.NM= AuditLogger: USER=3Dcf=09OPERATION=3DContainer Finished - Killed=09TARGET= =3DContainerImpl=09RESULT=3DSUCCESS=09APPID=3Dapplication_1468841606343_012= 8=09CONTAINERID=3Dcontainer_e05_1468841606343_0128_01_000003 > 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.container.Container: Container container_e05_1468841606343_0= 128_01_000003 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE > 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.application.Application: Removing container_e05_146884160634= 3_0128_01_000003 from application application_1468841606343_0128 > 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.logaggregation.AppLogAggregatorImpl: Considering container c= ontainer_e05_1468841606343_0128_01_000003 for log-aggregation > 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.AuxServices: Got event CONTAINER_STOP for appId application_= 1468841606343_0128 > 2016-07-20 09:04:25,656 INFO org.apache.spark.network.yarn.YarnShuffleSer= vice: Stopping container container_e05_1468841606343_0128_01_000003 > 2016-07-20 09:04:27,703 INFO org.apache.hadoop.yarn.server.nodemanager.No= deStatusUpdaterImpl: Removed completed containers from NM context: [contain= er_e05_1468841606343_0128_01_000003] > 2016-07-20 09:04:28,584 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring = for container_e05_1468841606343_0128_01_000003 > 2016-07-20 09:04:28,594 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 7737 for container-id container_e05_1468841606343_0021_01_000002: 285.7 MB = of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used > 2016-07-20 09:04:28,603 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 046 for container-id container_e05_1468841606343_0128_01_000001: 356.7 MB o= f 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used > 2016-07-20 09:04:28,613 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 116 for container-id container_e05_1468841606343_0128_01_000002: 348.1 MB o= f 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used > 2016-07-20 09:04:31,623 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 2= 7737 for container-id container_e05_1468841606343_0021_01_000002: 285.7 MB = of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)