Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 578AC18ADF for ; Wed, 18 Nov 2015 16:16:26 +0000 (UTC) Received: (qmail 27870 invoked by uid 500); 18 Nov 2015 16:16:22 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 27754 invoked by uid 500); 18 Nov 2015 16:16:22 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 27744 invoked by uid 99); 18 Nov 2015 16:16:22 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Nov 2015 16:16:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id BF7F9C07D8 for ; Wed, 18 Nov 2015 16:16:21 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.249 X-Spam-Level: *** X-Spam-Status: No, score=3.249 tagged_above=-999 required=6.31 tests=[FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=3, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id XRKsCMee2waB for ; Wed, 18 Nov 2015 16:16:19 +0000 (UTC) Received: from moh3-ve1.go2.pl (moh3-ve2.go2.pl [193.17.41.86]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 5E79421270 for ; Wed, 18 Nov 2015 16:16:18 +0000 (UTC) Received: from moh3-ve1.go2.pl (unknown [10.0.0.157]) by moh3-ve1.go2.pl (Postfix) with ESMTP id DB22CAF6970 for ; Wed, 18 Nov 2015 17:16:11 +0100 (CET) Received: from o2.pl (unknown [10.0.0.202]) by moh3-ve1.go2.pl (Postfix) with SMTP for ; Wed, 18 Nov 2015 17:16:11 +0100 (CET) Subject: =?UTF-8?Q?RE:_Does_MapReduceApplicationMaster?= =?UTF-8?Q?_prevents_data_node_from_spawning_YarnChild=3F?= From: =?UTF-8?Q?darekg11?= To: user@hadoop.apache.org In-Reply-To: References: <31410a51.23de0f71.564b4dc9.b7ebf@tlen.pl> <2a695bb5.64727fe0.564b8115.3267f@tlen.pl> Mime-Version: 1.0 Message-ID: <4a9b4d41.cfb9cf5.564ca44b.79cb7@tlen.pl> Date: Wed, 18 Nov 2015 17:16:11 +0100 X-Originator: 91.150.147.194 Content-Type: multipart/alternative; boundary="==o2.pl-WebMail-2830723e.71166577.79cca==" --==o2.pl-WebMail-2830723e.71166577.79cca== Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank=20You=20very=20much=20if=20it=20is=20the=20case=20then=20Applicatio= nMaster=20alone=20takes=20alomst=20everything=20that=20my=20datanode=20ca= n=20offer=20-=20that=20would=20explain=20why=20task=20itself=20can't=20st= art.=20Thank=20You=20very=20much=20again. Dnia=2017=20listopada=202015=2020:40=20Bikas=20Saha=20<bikas@apache.or= g>=20napisa=C5=82(a): You=20can=20check=20for=20=E2=80=9Cyarn.app.mapreduce.am.resource.mb=E2=80= =9D=20in=20your=20configs.=20Its=20default=20is=201.5GB=20and=20other=20M= R=20task=20defaults=20are=201GB.=C2=A0From:=20darekg11=20[mailto:darekg11= @tlen.pl] Sent:=20Tuesday,=20November=2017,=202015=2011:34=20AM To:=20user@hadoop.apache.org Subject:=20RE:=20Does=20MapReduceApplicationMaster=20prevents=20data=20no= de=20from=20spawning=20YarnChild?=C2=A0Thank=20You=20very=20much,=20will=20= check=20but=20the=20problem=20is=20that=20machines=20aren't=20really=20po= werful=20-=20only=202GB=20of=20RAM=20and=202xCore=20each=20at=203.3GHz. Do=20You=20maybe=20know=20approx=20value=20of=20required=20resources=20fo= r=20AppMaster?=20And=20can=20we=20check=20current=20resource=20consumptio= n=20of=20AppMaster? Dnia=2017=20listopada=202015=2018:40=20Bikas=20Saha=20<bikas@apache.or= g>=20napisa=C5=82(a):No.=20In=20general=20App=20Masters=20and=20their=20= containers=20can=20be=20launched=20on=20any=20machine=20and=20both=20can=20= be=20launched=20on=20the=20same=20machine.=C2=A0If=20your=20case=20happen= s=20repeatedly=20then=20you=20could=20check=20the=20RM=20UI,=20while=20th= e=20job=20is=20running,=20to=20see=20the=20maximum=20resource=20on=20a=20= node=20manager=20and=20the=20resource=20currently=20assigned.=20Perhaps=20= your=20node=20managers=20don=E2=80=99t=20have=20enough=20resources=20to=20= run=20multiple=20containers?=C2=A0From:=20darekg11=20[mailto:darekg11@tle= n.pl] Sent:=20Tuesday,=20November=2017,=202015=207:55=20AM To:=20user@hadoop.apache.org Subject:=20Does=20MapReduceApplicationMaster=20prevents=20data=20node=20f= rom=20spawning=20YarnChild?=C2=A0Hello=20again=20dear=20users. Today=20I=20ran=20into=20following=20problem: My=20mini=20cluster=20consist=20of: 1=20NameNode=20and=202=20SlaveNodes. When=20I=20ran=20my=20MapReduce=20program=20written=20in=20java=20with=20= number=20of=20reducers=20equals=20to=20two. As=20the=20result=20on=20first=20SlaveNode=20I=20got=20MRAppMaster=20task= =20and=20only=20the=20second=20slave=20launched=20Yarn=20Child=20which=20= actually=20was=20producing=20output=20results. I=20understand=20that=20MRAppMaster=20is=20essential=20process=20repsonsi= ble=20for=20managing=20life=20of=20given=20task. And=20because=20of=20that=20single=20slave=20node=20can't=20launch=20MrAp= pMaster=20and=20Yarn=20Child=20at=20the=20same=20time=20or=20am=20I=20mis= understanding=20something?=C2=A0 --==o2.pl-WebMail-2830723e.71166577.79cca== Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank=20You=20very=20much=20if=20it=20is=20the=20case=20then=20Applicatio= nMaster=20alone=20takes=20alomst=20everything=20that=20my=20datanode=20ca= n=20offer=20-=20that=20would=20explain=20why=20task=20itself=20can't=20st= art.=20Thank=20You=20very=20much=20again.Dnia=201= 7=20listopada=202015=2020:40=20Bikas=20Saha=20<bikas@apache.org>=20= napisa=C5=82(a): You=20can=20check=20= for=20=E2=80=9Cyarn.app.mapreduce.am.resource.mb=E2=80=9D=20in=20your=20c= onfigs.=20Its=20default=20is=201.5GB=20and=20other=20MR=20task=20defaults= =20are=201GB.

=C2=A0

From:=20= darekg11=20[mailto:darekg11@tlen.pl]Sent:=20Tues= day,=20November=2017,=202015=2011:34=20AMTo:=20u= ser@hadoop.apache.orgSubject:=20RE:=20Does=20Map= ReduceApplicationMaster=20prevents=20data=20node=20from=20spawning=20Yarn= Child?

=C2=A0

Thank=20You= =20very=20much,=20will=20check=20but=20the=20problem=20is=20that=20machin= es=20aren't=20really=20powerful=20-=20only=202GB=20of=20RAM=20and=202xCor= e=20each=20at=203.3GHz.Do=20You=20maybe=20know=20approx=20value=20= of=20required=20resources=20for=20AppMaster?=20And=20can=20we=20check=20c= urrent=20resource=20consumption=20of=20AppMaster?= Dnia=2017=20listopada=202015=2018:40=20Bikas=20Saha=20<bikas@apache.org>=20n= apisa=C5=82(a):

No.=20In=20general=20= App=20Masters=20and=20their=20containers=20can=20be=20launched=20on=20any= =20machine=20and=20both=20can=20be=20launched=20on=20the=20same=20machine= .

=C2=A0

If=20your=20case=20h= appens=20repeatedly=20then=20you=20could=20check=20the=20RM=20UI,=20while= =20the=20job=20is=20running,=20to=20see=20the=20maximum=20resource=20on=20= a=20node=20manager=20and=20the=20resource=20currently=20assigned.=20Perha= ps=20your=20node=20managers=20don=E2=80=99t=20have=20enough=20resources=20= to=20run=20multiple=20containers?

=C2=A0

From:=20= darekg11=20[mailto:darekg11@tlen.pl]Sent:=20Tuesday= ,=20November=2017,=202015=207:55=20AMTo:=20user@hadoop.apache.o= rgSubject:=20Does=20MapReduceApplicationMaster=20pre= vents=20data=20node=20from=20spawning=20YarnChild?

=C2=A0

Hello=20again=20dear=20users.Tod= ay=20I=20ran=20into=20following=20problem:My=20mini=20clu= ster=20consist=20of:1=20NameNode=20and=202=20SlaveNodes.<= br=20/>When=20I=20ran=20my=20MapReduce=20program=20written=20in=20java=20= with=20number=20of=20reducers=20equals=20to=20two.As=20the=20resu= lt=20on=20first=20SlaveNode=20I=20got=20MRAppMaster=20task=20and=20only=20= the=20second=20slave=20launched=20Yarn=20Child=20which=20actually=20was=20= producing=20output=20results.I=20understand=20that=20MRAp= pMaster=20is=20essential=20process=20repsonsible=20for=20managing=20life=20= of=20given=20task.And=20because=20of=20that=20single=20sl= ave=20node=20can't=20launch=20MrAppMaster=20and=20Yarn=20Child=20at=20the= =20same=20time=20or=20am=20I=20misunderstanding=20something?

=C2=A0

--==o2.pl-WebMail-2830723e.71166577.79cca==--