hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "S.L" <simpleliving...@gmail.com>
Subject Re: Unable to change the virtual memory to be more than the default 2.1 GB
Date Sun, 05 Jan 2014 17:41:34 GMT
Vinod,

Thanks for your reply.

1. If I understand you correct you are asking me to change the memory
allocation for each map and reduce tasks , isnt this related to the
physical memory which is not an issue(with in limits) in my application ?
The problem I am facing is with the virtual memory.

2. You are right I am spawning shells , but immediately closing them after
each request in the map task.  Why would virtual memory increase and the
physical memory stay the same , what might be the causes that would make
this happen in YARN ? How can I keep it with in manageable limits ?

If I run the same task as a stand alone program it works fine , obviously
because its not a memory leak kind of a scenario affecting the physical
memory.



On Thu, Jan 2, 2014 at 1:14 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> You need to change the application configuration itself to tell YARN that
> each task needs more than the default. I see that this is a mapreduce app,
> so you have to change the per-application configuration:
> mapreduce.map.memory.mb and mapreduce.reduce.memory.mb in either
> mapred-site.xml or via the command line.
>
> Side notes: Seems like you are spawning lots of shells under your mapper
> and YARN's NodeManager is detecting that the total virtual memory usage is
> 14.5GB. You may want to reduce that number of shells, lest the OS itself
> might kill your tasks depend on the system configuration.
>
> Thanks,
> +Vinod
>
> On Jan 1, 2014, at 7:50 PM, S.L <simpleliving016@gmail.com> wrote:
>
> Hello Folks,
>
> I am running hadoop 2.2 in a pseudo-distributed mode on a laptop with 8GB
> RAM.
>
> Whenever I submit a job I get an error that says that the that the virtual
> memory usage exceeded , like below.
>
> I have changed the ratio yarn.nodenamager.vmem-pmem-ratio in yarn-site.xml
> to 10 , however the virtual memory is not getting increased more than 2.1
> GB , as can been seen in the error message below and the container is being
> killed.
>
> Can some one please let me know if there is any other setting that needs
> to be changed ? Thanks in advance!
>
> *Error Message :*
>
> INFO mapreduce.Job: Task Id : attempt_1388632710048_0009_m_000000_2,
> Status : FAILED
> Container [pid=12013,containerID=container_1388632710048_0009_01_000004]
> is running beyond virtual memory limits. Current usage: 544.9 MB of 1 GB
> physical memory used; 14.5 GB of 2.1 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_1388632710048_0009_01_000004 :
>     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>     |- 12077 12018 12013 12013 (phantomjs) 16 2 1641000960 6728
> /usr/local/bin/phantomjs --webdriver=15358
> --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
>
>     |- 12013 882 12013 12013 (bash) 1 0 108650496 305 /bin/bash -c
> /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN  -Xmx200m
> -Djava.io.tmpdir=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_000004
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 127.0.0.1 56498
> attempt_1388632710048_0009_m_000000_2 4
> 1>/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_000004/stdout
> 2>/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_000004/stderr
>
>     |- 12075 12018 12013 12013 (phantomjs) 17 1 1615687680 6539
> /usr/local/bin/phantomjs --webdriver=29062
> --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
>
>     |- 12074 12018 12013 12013 (phantomjs) 16 2 1641000960 6727
> /usr/local/bin/phantomjs --webdriver=5958
> --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
>
>     |- 12073 12018 12013 12013 (phantomjs) 17 2 1641000960 6732
> /usr/local/bin/phantomjs --webdriver=31836
> --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
>
>     |- 12090 12018 12013 12013 (phantomjs) 16 2 1615687680 6538
> /usr/local/bin/phantomjs --webdriver=24519
> --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
>
>     |- 12072 12018 12013 12013 (phantomjs) 16 1 1641000960 6216
> /usr/local/bin/phantomjs --webdriver=10175
> --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
>
>     |- 12091 12018 12013 12013 (phantomjs) 17 1 1615687680 6036
> /usr/local/bin/phantomjs --webdriver=5043
> --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
>
>     |- 12018 12013 12013 12013 (java) 996 41 820924416 79595
> /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx200m
> -Djava.io.tmpdir=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_000004
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 127.0.0.1 56498
> attempt_1388632710048_0009_m_000000_2 4
>     |- 12078 12018 12013 12013 (phantomjs) 16 3 1615687680 6545
> /usr/local/bin/phantomjs --webdriver=12650
> --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
>
>     |- 12079 12018 12013 12013 (phantomjs) 17 2 1642020864 7542
> /usr/local/bin/phantomjs --webdriver=18444
> --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
>
>
> Container killed on request. Exit code is 143
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
View raw message