gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kam Kasravi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high
Date Wed, 31 Aug 2016 14:36:20 GMT

    [ https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452397#comment-15452397
] 

Kam Kasravi commented on GEARPUMP-185:
--------------------------------------

The problem (and resolution) was that the services process CLASSPATH did not include gear.conf
(gear.conf was not copied into the conf subdirectory) when this process was installed in TAP.
Once gear.conf was copied into this directory everything works as expected where the gear.conf
used to build gearpump commands (output/target/pack/bin) is used by all commands. I think
a reasonable improvement is a WARNING emitted to console if gear.conf is not found.

> Yarn kills Apache Gearpump's worker if the memory usage is too high
> -------------------------------------------------------------------
>
>                 Key: GEARPUMP-185
>                 URL: https://issues.apache.org/jira/browse/GEARPUMP-185
>             Project: Apache Gearpump
>          Issue Type: Bug
>         Environment: yarn
>            Reporter: Piotr Grabuszynski
>            Assignee: Kam Kasravi
>             Fix For: 0.8.0
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_000003 :
> 	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)
RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> 	|- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840 /usr/java/jdk1.8.0_72/jre/bin/java
-Dgearpump.username=vcap -Dgearpump.applicationId=1 -Dgearpump.executorId=0 -Dgearpump.master.starttime=2016-07-20-08-36
-Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf
...
> 	|- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626 /usr/java/jdk1.8.0_72/jre/bin/java
-Dgearpump.username=vcap -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 -Dgearpump.master.starttime=2016-07-20-08-36
-Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf
...
> 	|- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c /usr/java/jdk1.8.0_72/bin/java
 -Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
...
> 	|- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/stderr

> 	|- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249 /usr/java/jdk1.8.0_72/bin/java
-Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
...
> 	|- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356 /usr/java/jdk1.8.0_72/jre/bin/java
-Dgearpump.username=vcap -Dgearpump.applicationId=2 -Dgearpump.executorId=0 -Dgearpump.master.starttime=2016-07-20-08-36
-Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> 	|- 6360 2164 2159 2159 (java) 966 58 5845979136 54720 /usr/java/jdk1.8.0_72/jre/bin/java
-Dgearpump.username=vcap -Dgearpump.applicationId=2 -Dgearpump.executorId=-1 -Dgearpump.master.starttime=2016-07-20-08-36
-Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors' processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280  5.3  3.1 5406960 261356 pts/13 Sl+  09:28   0:10 /usr/lib/jvm/java-8-oracle/bin/java
-server -Djava.net.preferIPv4Stack=true -DlogFilename=master -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
-Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417  7.7  1.8 5392212 154728 pts/27 Sl+  09:29   0:05 /usr/lib/jvm/java-8-oracle/bin/java
-server -Djava.net.preferIPv4Stack=true -DlogFilename=worker -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
-Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0  3.1 4572132 255776 pts/27 Sl+  09:33   0:11 /usr/lib/jvm/java-8-oracle/jre/bin/java
-server -Xms512M -Xmx1024M -Xss1M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 -Djava.rmi.server.hostname=localhost
-Dgearpump.username=GER\pgrabusz -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761  132  5.2 7268592 427200 pts/27 Sl+  09:33   2:31 /usr/lib/jvm/java-8-oracle/jre/bin/java
-server -Xms512M -Xmx1024M -Xss1M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 [...] -Dgearpump.applicationId=1
-Dgearpump.executorId=0 ...
> {code}
> h3. Steps to Reproduce
> To generate trafic I've used such application:
> {code}
>                   kafka source
>                     /              \
>          kafka sink         hbase sink
> {code}
> Where kafka source write to the same topic as kafka source reads.
> Send some message to topic.
> Worker will disappear after few minutes (1-40 min depends on how much memory worker has);
> h3. Actual Results
> Worker has been killed
> h3. Additional information
> Increasing value of env WORKERS_MEMORY_LIMIT in gearpump-broker doesn't work. It increases
only running time.
> It might be a problem with some java properties (hip size or sth). It is also possible
there's some memory lick f.e. in hbase sink.
> Bigger piece of YARN log:
> {code}
> 2016-07-20 09:04:22,217 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 27737 for container-id container_e05_1468841606343_0021_01_000002:
285.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:22,270 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 2046 for container-id container_e05_1468841606343_0128_01_000001:
356.7 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:22,317 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 2159 for container-id container_e05_1468841606343_0128_01_000003:
2.0 GB of 2 GB physical memory used; 26.8 GB of 4.2 GB virtual memory used
> 2016-07-20 09:04:22,378 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 2116 for container-id container_e05_1468841606343_0128_01_000002:
348.0 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,424 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 27737 for container-id container_e05_1468841606343_0021_01_000002:
285.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,494 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 2046 for container-id container_e05_1468841606343_0128_01_000001:
356.7 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,538 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 2159 for container-id container_e05_1468841606343_0128_01_000003:
2.0 GB of 2 GB physical memory used; 26.8 GB of 4.2 GB virtual memory used
> 2016-07-20 09:04:25,539 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Process tree for container: container_e05_1468841606343_0128_01_000003 has processes older
than 1 iteration running over the configured limit. Limit=2147483648, current usage = 2167832576
> 2016-07-20 09:04:25,547 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Container [pid=2159,containerID=container_e05_1468841606343_0128_01_000003] is running beyond
physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 26.8 GB of 4.2
GB virtual memory used. Killing container.
> Dump of the process-tree for container_e05_1468841606343_0128_01_000003 :
> 	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)
RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> 	|- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840 /usr/java/jdk1.8.0_72/jre/bin/java
-Dgearpump.username=vcap -Dgearpump.applicationId=1 -Dgearpump.executorId=0 -Dgearpump.master.starttime=2016-07-20-08-36
-Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf
-cp conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> 	|- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626 /usr/java/jdk1.8.0_72/jre/bin/java
-Dgearpump.username=vcap -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 -Dgearpump.master.starttime=2016-07-20-08-36
-Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf
-cp conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> 	|- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c /usr/java/jdk1.8.0_72/bin/java
 -Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/lib/*
-Dgearpump.cluster.masters.0=cdh-worker-2.node.envname.consul:54946 -Dgearpump.log.daemon.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
-Dgearpump.worker-resource-manager-container-id=container_e05_1468841606343_0128_01_000003
-Dgearpump.home=/data1/hadoop/yarn/nm/usercache/cf/appcache/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/pack/gearpump-2.11-0.8.0
-Dgearpump.log.application.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
-Dgearpump.hostname=cdh-worker-2.node.envname.consul  io.gearpump.cluster.main.Worker  2>&1
| /usr/bin/tee -a /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/stderr

> 	|- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/stderr

> 	|- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249 /usr/java/jdk1.8.0_72/bin/java
-Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/lib/*
-Dgearpump.cluster.masters.0=cdh-worker-2.node.envname.consul:54946 -Dgearpump.log.daemon.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
-Dgearpump.worker-resource-manager-container-id=container_e05_1468841606343_0128_01_000003
-Dgearpump.home=/data1/hadoop/yarn/nm/usercache/cf/appcache/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/pack/gearpump-2.11-0.8.0
-Dgearpump.log.application.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
-Dgearpump.hostname=cdh-worker-2.node.envname.consul io.gearpump.cluster.main.Worker 
> 	|- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356 /usr/java/jdk1.8.0_72/jre/bin/java
-Dgearpump.username=vcap -Dgearpump.applicationId=2 -Dgearpump.executorId=0 -Dgearpump.master.starttime=2016-07-20-08-36
-Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/gearpump7648852439078424047.conf
-cp conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> 	|- 6360 2164 2159 2159 (java) 966 58 5845979136 54720 /usr/java/jdk1.8.0_72/jre/bin/java
-Dgearpump.username=vcap -Dgearpump.applicationId=2 -Dgearpump.executorId=-1 -Dgearpump.master.starttime=2016-07-20-08-36
-Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/gearpump7311147404608349128.conf
-cp conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib
> 2016-07-20 09:04:25,548 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_e05_1468841606343_0128_01_000003 transitioned from RUNNING to KILLING
> 2016-07-20 09:04:25,548 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:25,548 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Removed ProcessTree with root 2159
> 2016-07-20 09:04:25,584 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 2116 for container-id container_e05_1468841606343_0128_01_000002:
348.0 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,621 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Exit code from container container_e05_1468841606343_0128_01_000003 is : 143
> 2016-07-20 09:04:25,655 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_e05_1468841606343_0128_01_000003 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
> 2016-07-20 09:04:25,655 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Deleting absolute path : /data1/hadoop/yarn/nm/usercache/cf/appcache/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger:
USER=cf	OPERATION=Container Finished - Killed	TARGET=ContainerImpl	RESULT=SUCCESS	APPID=application_1468841606343_0128
CONTAINERID=container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_e05_1468841606343_0128_01_000003 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL
to DONE
> 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Removing container_e05_1468841606343_0128_01_000003 from application application_1468841606343_0128
> 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Considering container container_e05_1468841606343_0128_01_000003 for log-aggregation
> 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Got event CONTAINER_STOP for appId application_1468841606343_0128
> 2016-07-20 09:04:25,656 INFO org.apache.spark.network.yarn.YarnShuffleService: Stopping
container container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:27,703 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Removed completed containers from NM context: [container_e05_1468841606343_0128_01_000003]
> 2016-07-20 09:04:28,584 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Stopping resource-monitoring for container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:28,594 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 27737 for container-id container_e05_1468841606343_0021_01_000002:
285.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:28,603 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 2046 for container-id container_e05_1468841606343_0128_01_000001:
356.7 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:28,613 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 2116 for container-id container_e05_1468841606343_0128_01_000002:
348.1 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:31,623 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 27737 for container-id container_e05_1468841606343_0021_01_000002:
285.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message