gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kam Kasravi <kamkasr...@gmail.com>
Subject Re: [jira] [Commented] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high
Date Tue, 30 Aug 2016 20:17:38 GMT
resolution - the services application in TAP (MasterService.scala) creates
a process that runs the AppSubmitter - providing a classPath, and
GEARPUMP.HOME.
If the AppSubmitter is run where there is no gear.conf but there is a
geardefault.conf then gearpump.executor.vmargs is empty and passed as part
of the submission config.
Once it reaches the Worker - it will prioritize the submission config over
it's own config when running the executor.
This can happen when the services app is launched incorrectly - without
having a gear.conf but having a geardefault.conf (in this case
geardefault.conf came the core jar.

On Tue, Aug 30, 2016 at 5:36 AM, Kam Kasravi (JIRA) <jira@apache.org> wrote:

>
>     [ https://issues.apache.org/jira/browse/GEARPUMP-185?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=15448900#comment-15448900 ]
>
> Kam Kasravi commented on GEARPUMP-185:
> --------------------------------------
>
> i'm able to confirm that the missing vmargs is related to the user
> argument. Under the applications section of the dashboard UI - the vcap
> users application's config had no vmargs. Launching a new application
> (which ran as yarn) had a config that contained a valid vmargs for the
> executor
>
> > Yarn kills Apache Gearpump's worker if the memory usage is too high
> > -------------------------------------------------------------------
> >
> >                 Key: GEARPUMP-185
> >                 URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> >             Project: Apache Gearpump
> >          Issue Type: Bug
> >         Environment: yarn
> >            Reporter: Piotr Grabuszynski
> >            Assignee: Kam Kasravi
> >             Fix For: 0.8.0
> >
> >
> > h3. Description
> > If the traffic is to big (memory usage is height) Yarn kills Gearpump's
> worker.
> > Clean YARN log (only the part with processes, bigger piece below):
> > {code}
> > Dump of the process-tree for container_e05_1468841606343_0128_01_000003
> :
> >       |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> >       |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36 -Dgearpump.log.application.dir=logs
> -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/
> gearpump6797336735445779094.conf ...
> >       |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36 -Dgearpump.log.application.dir=logs
> -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/
> gearpump941454978115257558.conf ...
> >       |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java  -Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/
> conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.
> 11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/
> cloudera-scm-agent/process/195-yarn-NODEMANAGER ...
> >       |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_
> e05_1468841606343_0128_01_000003/stderr
> >       |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/
> conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.
> 11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/
> cloudera-scm-agent/process/195-yarn-NODEMANAGER ...
> >       |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36 -Dgearpump.log.application.dir=logs
> -Djava.net.preferIPv4Stack=true ...
> >       |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36 -Dgearpump.log.application.dir=logs
> -Djava.net.preferIPv4Stack=true ...
> > {code}
> > *As we can see there are no "-Xmx" or "-Xsx" parameters near executors'
> processes.*
> > In standalone mode, we can see both:
> > {code}
> > // MASTER
> > GER\pgr+ 48280  5.3  3.1 5406960 261356 pts/13 Sl+  09:28   0:10
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=master -Dgearpump.home=/home/local/GER/pgrabusz/Documents/
> gearpump-sources/gearpump-2.11-0.8.0 -Djava.rmi.server.hostname=localhost
> ...
> > // WORKER
> > GER\pgr+ 48417  7.7  1.8 5392212 154728 pts/27 Sl+  09:29   0:05
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=worker -Dgearpump.home=/home/local/GER/pgrabusz/Documents/
> gearpump-sources/gearpump-2.11-0.8.0 -Djava.rmi.server.hostname=localhost
> ...
> > // APP MASTER
> > GER\pgr+ 48707 15.0  3.1 4572132 255776 pts/27 Sl+  09:33   0:11
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:
> CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> > // APP EXECUTOR 0
> > GER\pgr+ 48761  132  5.2 7268592 427200 pts/27 Sl+  09:33   2:31
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:
> CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 [...]
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 ...
> > {code}
> > h3. Steps to Reproduce
> > To generate trafic I've used such application:
> > {code}
> >                   kafka source
> >                     /              \
> >          kafka sink         hbase sink
> > {code}
> > Where kafka source write to the same topic as kafka source reads.
> > Send some message to topic.
> > Worker will disappear after few minutes (1-40 min depends on how much
> memory worker has);
> > h3. Actual Results
> > Worker has been killed
> > h3. Additional information
> > Increasing value of env WORKERS_MEMORY_LIMIT in gearpump-broker doesn't
> work. It increases only running time.
> > It might be a problem with some java properties (hip size or sth). It is
> also possible there's some memory lick f.e. in hbase sink.
> > Bigger piece of YARN log:
> > {code}
> > 2016-07-20 09:04:22,217 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 27737 for container-id container_e05_1468841606343_0021_01_000002:
> 285.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> > 2016-07-20 09:04:22,270 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 2046 for container-id container_e05_1468841606343_0128_01_000001:
> 356.7 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used
> > 2016-07-20 09:04:22,317 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 2159 for container-id container_e05_1468841606343_0128_01_000003:
> 2.0 GB of 2 GB physical memory used; 26.8 GB of 4.2 GB virtual memory used
> > 2016-07-20 09:04:22,378 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 2116 for container-id container_e05_1468841606343_0128_01_000002:
> 348.0 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> > 2016-07-20 09:04:25,424 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 27737 for container-id container_e05_1468841606343_0021_01_000002:
> 285.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> > 2016-07-20 09:04:25,494 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 2046 for container-id container_e05_1468841606343_0128_01_000001:
> 356.7 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used
> > 2016-07-20 09:04:25,538 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 2159 for container-id container_e05_1468841606343_0128_01_000003:
> 2.0 GB of 2 GB physical memory used; 26.8 GB of 4.2 GB virtual memory used
> > 2016-07-20 09:04:25,539 WARN org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Process tree
> for container: container_e05_1468841606343_0128_01_000003 has processes
> older than 1 iteration running over the configured limit. Limit=2147483648,
> current usage = 2167832576
> > 2016-07-20 09:04:25,547 WARN org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Container
> [pid=2159,containerID=container_e05_1468841606343_0128_01_000003] is
> running beyond physical memory limits. Current usage: 2.0 GB of 2 GB
> physical memory used; 26.8 GB of 4.2 GB virtual memory used. Killing
> container.
> > Dump of the process-tree for container_e05_1468841606343_0128_01_000003
> :
> >       |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> >       |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36 -Dgearpump.log.application.dir=logs
> -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/
> gearpump6797336735445779094.conf -cp conf:pack/gearpump-2.11-0.8.0/
> conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:
> pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.
> jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.
> 0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-
> metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/
> scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-
> log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-
> 1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.
> 2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-
> combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/
> akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/
> scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/
> lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-
> agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-
> library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/
> gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/
> gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/
> gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/
> gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:
> pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:
> pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/
> gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:
> pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-
> collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/
> gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.
> cloudera.YARN:/var/run/cloudera-scm-agent/process/
> 195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:
> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/
> opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/
> opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.
> 1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/
> parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-
> scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.
> jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.
> 1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/
> CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-
> sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-
> cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:
> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> >       |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36 -Dgearpump.log.application.dir=logs
> -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/
> gearpump941454978115257558.conf -cp conf:pack/gearpump-2.11-0.8.0/
> conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:
> pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.
> jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.
> 0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-
> metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/
> scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-
> log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-
> 1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.
> 2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-
> combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/
> akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/
> scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/
> lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-
> agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-
> library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/
> gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/
> gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/
> gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/
> gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:
> pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:
> pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/
> gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:
> pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-
> collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/
> gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.
> cloudera.YARN:/var/run/cloudera-scm-agent/process/
> 195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:
> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/
> opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/
> opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.
> 1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/
> parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-
> scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.
> jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.
> 1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/
> CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-
> sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-
> cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:
> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> >       |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java  -Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/
> conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.
> 11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/
> cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/*:/
> opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/
> lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop-hdfs/lib/*:/opt/cloudera/
> parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/*:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/lib/*
> -Dgearpump.cluster.masters.0=cdh-worker-2.node.envname.consul:54946
> -Dgearpump.log.daemon.dir=/var/log/hadoop-yarn/container/
> application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
> -Dgearpump.worker-resource-manager-container-id=
> container_e05_1468841606343_0128_01_000003 -Dgearpump.home=/data1/hadoop/
> yarn/nm/usercache/cf/appcache/application_1468841606343_
> 0128/container_e05_1468841606343_0128_01_000003/pack/gearpump-2.11-0.8.0
> -Dgearpump.log.application.dir=/var/log/hadoop-yarn/container/application_
> 1468841606343_0128/container_e05_1468841606343_0128_01_000003
> -Dgearpump.hostname=cdh-worker-2.node.envname.consul
> io.gearpump.cluster.main.Worker  2>&1 | /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_
> e05_1468841606343_0128_01_000003/stderr
> >       |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_
> e05_1468841606343_0128_01_000003/stderr
> >       |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp conf:pack/gearpump-2.11-0.8.0/
> conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.
> 11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/
> cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/*:/
> opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/
> lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop-hdfs/lib/*:/opt/cloudera/
> parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/*:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/lib/*
> -Dgearpump.cluster.masters.0=cdh-worker-2.node.envname.consul:54946
> -Dgearpump.log.daemon.dir=/var/log/hadoop-yarn/container/
> application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
> -Dgearpump.worker-resource-manager-container-id=
> container_e05_1468841606343_0128_01_000003 -Dgearpump.home=/data1/hadoop/
> yarn/nm/usercache/cf/appcache/application_1468841606343_
> 0128/container_e05_1468841606343_0128_01_000003/pack/gearpump-2.11-0.8.0
> -Dgearpump.log.application.dir=/var/log/hadoop-yarn/container/application_
> 1468841606343_0128/container_e05_1468841606343_0128_01_000003
> -Dgearpump.hostname=cdh-worker-2.node.envname.consul
> io.gearpump.cluster.main.Worker
> >       |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36 -Dgearpump.log.application.dir=logs
> -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/
> gearpump7648852439078424047.conf -cp conf:pack/gearpump-2.11-0.8.0/
> conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:
> pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.
> jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.
> 0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-
> metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/
> scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-
> log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-
> 1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.
> 2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-
> combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/
> akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/
> scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/
> lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-
> agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-
> library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/
> gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/
> gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/
> gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/
> gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:
> pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:
> pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/
> gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:
> pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-
> collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/
> gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.
> cloudera.YARN:/var/run/cloudera-scm-agent/process/
> 195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:
> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/
> opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/
> opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.
> 1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/
> parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-
> scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.
> jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.
> 1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/
> CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-
> sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-
> cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:
> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> >       |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36 -Dgearpump.log.application.dir=logs
> -Djava.net.preferIPv4Stack=true -Dgearpump.config.file=/tmp/
> gearpump7311147404608349128.conf -cp conf:pack/gearpump-2.11-0.8.0/
> conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:
> pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.
> jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.
> 0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-
> metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/
> scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-
> log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-
> 1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.
> 2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-
> combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/
> akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/
> scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/
> lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-
> agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-
> library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/
> gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/
> gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/
> gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/
> gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:
> pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:
> pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/
> gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:
> pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-
> collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/
> gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.
> cloudera.YARN:/var/run/cloudera-scm-agent/process/
> 195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:
> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/
> opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/
> opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.
> 1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/
> parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-
> scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.
> jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.
> 1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/
> cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/
> hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/
> CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-
> sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.
> p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-
> 5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-
> cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.
> cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:
> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib
> > 2016-07-20 09:04:25,548 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.container.Container: Container
> container_e05_1468841606343_0128_01_000003 transitioned from RUNNING to
> KILLING
> > 2016-07-20 09:04:25,548 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up
> container container_e05_1468841606343_0128_01_000003
> > 2016-07-20 09:04:25,548 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Removed
> ProcessTree with root 2159
> > 2016-07-20 09:04:25,584 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 2116 for container-id container_e05_1468841606343_0128_01_000002:
> 348.0 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> > 2016-07-20 09:04:25,621 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Exit code from container container_e05_1468841606343_0128_01_000003 is :
> 143
> > 2016-07-20 09:04:25,655 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.container.Container: Container
> container_e05_1468841606343_0128_01_000003 transitioned from KILLING to
> CONTAINER_CLEANEDUP_AFTER_KILL
> > 2016-07-20 09:04:25,655 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Deleting absolute path : /data1/hadoop/yarn/nm/usercache/cf/appcache/
> application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
> > 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger:
> USER=cf OPERATION=Container Finished - Killed   TARGET=ContainerImpl
> RESULT=SUCCESS  APPID=application_1468841606343_0128
> CONTAINERID=container_e05_1468841606343_0128_01_000003
> > 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.container.Container: Container
> container_e05_1468841606343_0128_01_000003 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> > 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.application.Application: Removing
> container_e05_1468841606343_0128_01_000003 from application
> application_1468841606343_0128
> > 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
> Considering container container_e05_1468841606343_0128_01_000003 for
> log-aggregation
> > 2016-07-20 09:04:25,656 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for
> appId application_1468841606343_0128
> > 2016-07-20 09:04:25,656 INFO org.apache.spark.network.yarn.YarnShuffleService:
> Stopping container container_e05_1468841606343_0128_01_000003
> > 2016-07-20 09:04:27,703 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Removed completed containers from NM context: [container_e05_1468841606343_
> 0128_01_000003]
> > 2016-07-20 09:04:28,584 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping
> resource-monitoring for container_e05_1468841606343_0128_01_000003
> > 2016-07-20 09:04:28,594 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 27737 for container-id container_e05_1468841606343_0021_01_000002:
> 285.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> > 2016-07-20 09:04:28,603 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 2046 for container-id container_e05_1468841606343_0128_01_000001:
> 356.7 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used
> > 2016-07-20 09:04:28,613 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 2116 for container-id container_e05_1468841606343_0128_01_000002:
> 348.1 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> > 2016-07-20 09:04:31,623 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 27737 for container-id container_e05_1468841606343_0021_01_000002:
> 285.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used
> > {code}
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message