Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BCC10181C9 for ; Tue, 17 Nov 2015 19:40:16 +0000 (UTC) Received: (qmail 41049 invoked by uid 500); 17 Nov 2015 19:40:12 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 40818 invoked by uid 500); 17 Nov 2015 19:40:11 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 40684 invoked by uid 99); 17 Nov 2015 19:40:11 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2015 19:40:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 5AE711A0AB1 for ; Tue, 17 Nov 2015 19:40:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.999 X-Spam-Level: *** X-Spam-Status: No, score=3.999 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_MSPIKE_H2=-0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 8ryj_nEabihv for ; Tue, 17 Nov 2015 19:40:10 +0000 (UTC) Received: from BLU004-OMC2S5.hotmail.com (blu004-omc2s5.hotmail.com [65.55.111.80]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 42C6520271 for ; Tue, 17 Nov 2015 19:40:09 +0000 (UTC) Received: from BLU403-EAS221 ([65.55.111.72]) by BLU004-OMC2S5.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Tue, 17 Nov 2015 11:40:08 -0800 X-TMN: [6j7XEWqzh5aFTGkUgYx/m7f3AE8bFrkn] X-Originating-Email: [bikas@apache.org] Message-ID: From: Bikas Saha To: References: <31410a51.23de0f71.564b4dc9.b7ebf@tlen.pl> <2a695bb5.64727fe0.564b8115.3267f@tlen.pl> In-Reply-To: <2a695bb5.64727fe0.564b8115.3267f@tlen.pl> Subject: RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Date: Tue, 17 Nov 2015 11:40:07 -0800 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_001A_01D1212C.B5AA75A0" X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQABAgMEWr5F42qo/+P+3ShbvfonzgCVnNF4AGTGQn6iOQftcA== Content-Language: en-us Sender: X-OriginalArrivalTime: 17 Nov 2015 19:40:08.0560 (UTC) FILETIME=[C41A4F00:01D1216F] ------=_NextPart_000_001A_01D1212C.B5AA75A0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable You can check for =E2=80=9Cyarn.app.mapreduce.am.resource.mb=E2=80=9D in = your configs. Its default is 1.5GB and other MR task defaults are 1GB. =20 From: darekg11 [mailto:darekg11@tlen.pl]=20 Sent: Tuesday, November 17, 2015 11:34 AM To: user@hadoop.apache.org Subject: RE: Does MapReduceApplicationMaster prevents data node from = spawning YarnChild? =20 Thank You very much, will check but the problem is that machines aren't = really powerful - only 2GB of RAM and 2xCore each at 3.3GHz. Do You maybe know approx value of required resources for AppMaster? And = can we check current resource consumption of AppMaster? Dnia 17 listopada 2015 18:40 Bikas Saha > napisa=C5=82(a): No. In general App Masters and their containers can be launched on any = machine and both can be launched on the same machine. =20 If your case happens repeatedly then you could check the RM UI, while = the job is running, to see the maximum resource on a node manager and = the resource currently assigned. Perhaps your node managers = don=E2=80=99t have enough resources to run multiple containers? =20 From: darekg11 [mailto:darekg11@tlen.pl] Sent: Tuesday, November 17, 2015 7:55 AM To: user@hadoop.apache.org =20 Subject: Does MapReduceApplicationMaster prevents data node from = spawning YarnChild? =20 Hello again dear users. Today I ran into following problem: My mini cluster consist of: 1 NameNode and 2 SlaveNodes. When I ran my MapReduce program written in java with number of reducers = equals to two. As the result on first SlaveNode I got MRAppMaster task and only the = second slave launched Yarn Child which actually was producing output = results. I understand that MRAppMaster is essential process repsonsible for = managing life of given task. And because of that single slave node can't launch MrAppMaster and Yarn = Child at the same time or am I misunderstanding something? =20 ------=_NextPart_000_001A_01D1212C.B5AA75A0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

You can check for =E2=80=9Cyarn.app.mapreduce.am.resource.mb=E2=80=9D = in your configs. Its default is 1.5GB and other MR task defaults are = 1GB.

 

From:<= /b> = darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November = 17, 2015 11:34 AM
To: = user@hadoop.apache.org
Subject: RE: Does = MapReduceApplicationMaster prevents data node from spawning = YarnChild?

 

Thank You very much, will check but the = problem is that machines aren't really powerful - only 2GB of RAM and = 2xCore each at 3.3GHz.
Do You maybe know approx value of required = resources for AppMaster? And can we check current resource consumption = of AppMaster?


Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org> = napisa=C5=82(a):

No. In general App Masters and their containers can be launched on any = machine and both can be launched on the same = machine.

 

If your case happens repeatedly then you could check the RM UI, while = the job is running, to see the maximum resource on a node manager and = the resource currently assigned. Perhaps your node managers = don=E2=80=99t have enough resources to run multiple = containers?

 

From:<= /strong> darekg11 [mailto:darekg11@tlen.pl]
= Sent: = Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does = MapReduceApplicationMaster prevents data node from spawning = YarnChild?

 <= /o:p>

Hello again = dear users.

Today I ran into following problem:

My mini = cluster consist of:
1 NameNode and 2 SlaveNodes.

When I ran my = MapReduce program written in java with number of reducers equals to = two.
As the result on first SlaveNode I got MRAppMaster task and only = the second slave launched Yarn Child which actually was producing output = results.

I understand that MRAppMaster is essential process = repsonsible for managing life of given task.

And because of that = single slave node can't launch MrAppMaster and Yarn Child at the same = time or am I misunderstanding = something?

 

------=_NextPart_000_001A_01D1212C.B5AA75A0--