flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexis Gendronneau <a.gendronn...@gmail.com>
Subject Re: yarn kill container due to running beyond physical memory limits [ How can i debug memory issue ]
Date Wed, 01 Jun 2016 00:05:58 GMT
Hello,

How many memory your yarn containers are configured to have ? This error
may be due to running a flink on yarn cluster with more memory than you
have in containers. Could you check it, and maybe set containers memory to
a more suitable value ?

regards

2016-06-01 1:22 GMT+02:00 prateekarora <prateek.arora2k6@gmail.com>:

> Hi
>
> I am running flink 1.0.2 with Yarn .
>
> After running application for some time , Yarn kill my container due to
> running beyond physical memory limits .
>
> how can i debug memory issue ?
>
> below are the logs :
>
> Container container_1463184272818_0165_01_000012 is completed with
> diagnostics: Container
> [pid=19349,containerID=container_1463184272818_0165_01_000012] is running
> beyond physical memory limits. Current usage: 6.0 GB of 6 GB physical
> memory
> used; 9.1 GB of 12.6 GB virtual memory used. Killing container.
>
> Dump of the process-tree for container_1463184272818_0165_01_000012 :
>         |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>         |- 19356 19349 19349 19349 (java) 39350 9110 9711140864 1581168
> /usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m
> -XX:MaxDirectMemorySize=4608m
> -Djava.library.path=/home/nativelibraries/native_lib/
>
> -Dlog.file=/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.log
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
>         |- 19349 19345 19349 19349 (bash) 0 0 11456512 359 /bin/bash -c
> /usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m
> -XX:MaxDirectMemorySize=4608m
> -Djava.library.path=/home/nativelibraries/native_lib/
>
> -Dlog.file=/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.log
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
>
> /var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.out
> 2>
>
> /var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.err
>
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/yarn-kill-container-due-to-running-beyond-physical-memory-limits-How-can-i-debug-memory-issue-tp7296.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>



-- 
Alexis Gendronneau

alexis.gendronneau@corp.ovh.com
a.gendronneau@gmail.com

Mime
View raw message