Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E51511000B for ; Tue, 25 Feb 2014 16:04:09 +0000 (UTC) Received: (qmail 56743 invoked by uid 500); 25 Feb 2014 16:04:02 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 56374 invoked by uid 500); 25 Feb 2014 16:04:01 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 56363 invoked by uid 99); 25 Feb 2014 16:04:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Feb 2014 16:04:00 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of patrick.boenzli@soom-it.ch designates 194.126.200.49 as permitted sender) Received: from [194.126.200.49] (HELO server39.cyon.ch) (194.126.200.49) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Feb 2014 16:03:55 +0000 Received: from 220-24.107-92.cust.bluewin.ch ([92.107.24.220]:58822 helo=[10.0.1.72]) by server39.cyon.ch with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80.1) (envelope-from ) id 1WIKTk-002xM4-5z for user@hadoop.apache.org; Tue, 25 Feb 2014 17:03:33 +0100 From: Patrick Boenzli Content-Type: multipart/alternative; boundary="Apple-Mail=_11177E7B-FE76-456D-A8F6-ADA1D603D220" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: Hadoop YARN 2.2.0 Streaming Memory Limitation Date: Tue, 25 Feb 2014 17:03:00 +0100 References: <9165A19D-D286-4F0F-B410-D4C61C58978D@soom-it.ch> To: user@hadoop.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1827) X-OutGoing-Spam-Status: No, score=-1.0 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server39.cyon.ch X-AntiAbuse: Original Domain - hadoop.apache.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - soom-it.ch X-Get-Message-Sender-Via: server39.cyon.ch: authenticated_id: patrick.boenzli@soom-it.ch X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_11177E7B-FE76-456D-A8F6-ADA1D603D220 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hi Arun, hi all, thanks a lot for your input. we got it to run correctly, although not = exactly the solution you proposed, but it=92s close: the main error we made is that on a yarn controller node the memory = footprint must be set differently than on a hadoop worker node. = following rule of thumb seems to apply in our setup: master:=20 mapreduce.map.memory.mb =3D 1/3 of yarn.nodemanager.resource.memory-mb worker: mapreduce.map.memory.mb =3D 1/2 of yarn.nodemanager.resource.memory-mb for both cases we set: mapreduce.map.child.java.opts=3D=93Xmx 1024=94 or about 1/4 of total = memory. The reason for this behaviour is that the yarn controller spawns 2 = subprocesses, while all worker spawn only 1 subprocess: - on master: java MRAppMaster and YarnChild (which spawns the mapper) - on workers: YarnChild (which spawns the mapper) Now everything works smoothly. Thanks a lot again! Patrick On 24 Feb 2014, at 23:49, Arun C Murthy wrote: > Can you pls try with mapreduce.map.memory.mb =3D 5124 & = mapreduce.map.child.java.opts=3D"-Xmx1024" ?=20 >=20 > This way the map jvm gets 1024 and 4G is available for the container. >=20 > Hope that helps. >=20 > Arun >=20 > On Feb 24, 2014, at 1:27 AM, Patrick Boenzli = wrote: >=20 >> hello hadoop-users! >>=20 >> We are currently facing a frustrating hadoop streaming memory = problem. our setup: >>=20 >> our compute nodes have about 7 GB of RAM >> hadoop streaming starts a bash script wich uses about 4 GB of RAM >> therefore it is only possible to start one and only one task per node >> out of the box each hadoop instance starts about 7 hadoop containers = with default hadoop settings. each hadoop task forks a bash script that = need about 4 GB of RAM, the first fork works, all following fail because = they run out of memory. so what we are looking for is to limit the = number of containers to only one. so what we found on the internet: >>=20 >> yarn.scheduler.maximum-allocation-mb and mapreduce.map.memory.mb is = set to values such that there is at most one container. this means, = mapreduce.map.memory.mb must be more than half of the maximum memory = (otherwise there will be multiple containers). >> done right, this gives us one container per node. but it produces a = new problem: since our java process is now using at least half of the = max memory, our child (bash) process we fork will inherit the parent = memory footprint and since the memory used by our parent was more than = half of total memory, we run out of memory again. if we lower the map = memory, hadoop will allocate 2 containers per node, which will run out = of memory too. >>=20 >> since this problem is a blocker in our current project we are = evaluating adapting the source code to solve this issue. as a last = resort. any ideas on this are very much welcome. >>=20 >> we would be very happy for any help offered!=20 >> Thanks! >>=20 >>=20 >> PS: We asked this question also on stackoverflow three days ago = (http://stackoverflow.com/questions/21933937/hadoop-2-2-0-streaming-memory= -limitation). no answer yet. If there should be any answers in one of = the forms we will sync the answers. >=20 > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ >=20 >=20 >=20 > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or = entity to which it is addressed and may contain information that is = confidential, privileged and exempt from disclosure under applicable = law. If the reader of this message is not the intended recipient, you = are hereby notified that any printing, copying, dissemination, = distribution, disclosure or forwarding of this communication is strictly = prohibited. If you have received this communication in error, please = contact the sender immediately and delete it from your system. Thank = You. --Apple-Mail=_11177E7B-FE76-456D-A8F6-ADA1D603D220 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252

Hi Arun,
hi = all,

thanks a lot for your input. we got it to = run correctly, although not exactly the solution you proposed, but it=92s = close:


the main error we made is = that on a yarn controller node the memory footprint must be set = differently than on a hadoop worker node. following rule of thumb seems = to apply in our = setup:

master: 
mapreduce.map.memo= ry.mb =3D 1/3 of = yarn.nodemanager.resource.memory-mb

worker:
=
mapreduce.map.memory.mb =3D 1/2 of = yarn.nodemanager.resource.memory-mb


for both cases we = set:
mapreduce.map.child.java.opts=3D=93Xmx 1024=94 or about = 1/4 of total = memory.



The = reason for this behaviour is that the yarn controller spawns 2 = subprocesses, while all worker spawn only 1 subprocess:
- on = master: java MRAppMaster and YarnChild (which spawns the = mapper)
- on workers: YarnChild (which spawns the = mapper)

Now everything works smoothly. Thanks a = lot again!
Patrick




On 24 Feb 2014, at 23:49, Arun C Murthy <acm@hortonworks.com> = wrote:

Can = you pls try with mapreduce.map.memory.mb =3D 5124 & = mapreduce.map.child.java.opts=3D"-Xmx1024" = ? 

This way the map jvm gets 1024 and 4G is = available for the container.

Hope that = helps.

Arun

On= Feb 24, 2014, at 1:27 AM, Patrick Boenzli <patrick.boenzli@soom-it.ch&= gt; wrote:

hello = hadoop-users!

We are currently facing a = frustrating hadoop streaming memory problem. our setup:

  • our = compute nodes have about 7 GB of RAM
  • hadoop streaming starts a bash script wich uses = about 4 GB of = RAM
  • therefore it = is only possible to start one and only one task per node

out of the box each = hadoop instance starts about 7 hadoop containers with default hadoop = settings. each hadoop task forks a bash script that need about 4 GB of = RAM, the first fork works, all following fail because they run out of = memory. so what we are looking for is to limit the number = of containers to only = one. so what we found on the internet:

  • yarn.scheduler.maximum-allocation-mb and mapreduce.map.memory.mb is set to = values such that there is at most one container. this means, mapreduce.map.memory.mb must = be more than = half of the maximum memory (otherwise there will be = multiple containers).

done right, this = gives us one container per node. but it produces a new problem: since = our java process is now using at least half of the max memory, our child = (bash) process we fork will inherit the parent memory footprint and = since the memory used by our parent was more than half of total = memory, we run out of = memory again. if we lower the map memory, hadoop will allocate = 2 containers per node, which will run out of memory too.

since this problem is a blocker in our current project = we are evaluating adapting the source code to solve this issue. as a = last resort. any ideas on this are very much welcome.

we would be very happy for any = help offered! 
Thanks!


PS: We asked this question = also on stackoverflow three days ago (http://stackoverflow.com/questions/21933937/hadoop-2-2= -0-streaming-memory-limitation). no answer yet. If there should be any answers in one of = the forms we will sync the = answers.

Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

=


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or = entity to which it is addressed and may contain information that is = confidential, privileged and exempt from disclosure under applicable = law. If the reader of this message is not the intended recipient, you = are hereby notified that any printing, copying, dissemination, = distribution, disclosure or forwarding of this communication is strictly = prohibited. If you have received this communication in error, please = contact the sender immediately and delete it from your system. Thank = You.

= --Apple-Mail=_11177E7B-FE76-456D-A8F6-ADA1D603D220--