Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
MIME-Version: 1.0
In-Reply-To: <A726D7DD-2323-4B93-A6CE-2B84E568468C@apache.org>
References: 
 <CAD1=LDopuagBgksJpFELMrbBy6O9f6Mr0YpZBKeTU-JpW1gOTg@mail.gmail.com>
	<A726D7DD-2323-4B93-A6CE-2B84E568468C@apache.org>
Date: Fri, 6 Nov 2015 17:24:54 +0530
Message-ID: 
 <CAD1=LDrnYmxc3AgObENM=XTnmJ=g-XP-um3uwPtS9htx4iduwg@mail.gmail.com>
Subject: Re: Max Parallel task executors
From: sandeep das <yarnhadoop@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a11c12fd66f202c0523dde943

--001a11c12fd66f202c0523dde943
Content-Type: text/plain; charset=UTF-8

Hi Varun,

I tried to increase this parameter but it did not increase number of
parallel tasks but if It is decreased then YARN reduces number of parallel
tasks. I'm bit puzzled why its not increasing more than 31 tasks even after
its value is increased.

Is there any other configuration as well which controls on how many maximum
tasks can execute in parallel?

Regards,
Sandeep

On Tue, Nov 3, 2015 at 7:29 PM, Varun Vasudev <vvasudev@apache.org> wrote:

> The number of parallel tasks that are run depends on the amount of memory
> and vcores on your machines and the amount of memory and vcores required by
> your mappers and reducers. The amount of memory can be set
> via yarn.nodemanager.resource.memory-mb(the default is 8G). The amount of
> vcores can be set via yarn.nodemanager.resource.cpu-vcores(the default is
> 8 vcores).
>
> -Varun
>
> From: sandeep das <yarnhadoop@gmail.com>
> Reply-To: <user@hadoop.apache.org>
> Date: Monday, November 2, 2015 at 3:56 PM
> To: <user@hadoop.apache.org>
> Subject: Max Parallel task executors
>
> Hi Team,
>
> I've a cloudera cluster of 4 nodes. Whenever i submit a job my only 31
> parallel tasks are executed whereas my machines have more CPU available but
> still YARN/AM does not create more task.
>
> Is there any configuration which I can change to start more MAP/REDUCER
> task in parallel?
>
> Each machine in my cluster has 24 CPUs.
>
> Regards,
> Sandeep
>

--001a11c12fd66f202c0523dde943
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi Varun,<br><br></div><div>I tried to increase =
this parameter but it did not increase number of parallel tasks but if It i=
s decreased then YARN reduces number of parallel tasks. I&#39;m bit puzzled=
 why its not increasing more than 31 tasks even after its value is increase=
d.<br><br></div><div>Is there any other configuration as well which control=
s on how many maximum tasks can execute in parallel?<br></div><div><br></di=
v>Regards,<br></div>Sandeep<br></div><div class=3D"gmail_extra"><br><div cl=
ass=3D"gmail_quote">On Tue, Nov 3, 2015 at 7:29 PM, Varun Vasudev <span dir=
=3D"ltr">&lt;<a href=3D"mailto:vvasudev@apache.org" target=3D"_blank">vvasu=
dev@apache.org</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" s=
tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div=
 style=3D"word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:=
Calibri,sans-serif"><div><div><div>The number of parallel tasks that are ru=
n depends on the amount of memory and vcores on your machines and the amoun=
t of memory and vcores required by your mappers and reducers. The amount of=
 memory can be set via=C2=A0yarn.nodemanager.resource.memory-mb(the default=
 is 8G). The amount of vcores can be set via<i>=C2=A0</i>yarn.nodemanager.r=
esource.cpu-vcores(the default is 8 vcores).</div><div><br></div><div>-Varu=
n</div><div><div></div></div></div></div><div><br></div><span><div style=3D=
"font-family:Calibri;font-size:12pt;text-align:left;color:black;BORDER-BOTT=
OM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;=
PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PAD=
DING-TOP:3pt"><span style=3D"font-weight:bold">From: </span> sandeep das &l=
t;<a href=3D"mailto:yarnhadoop@gmail.com" target=3D"_blank">yarnhadoop@gmai=
l.com</a>&gt;<br><span style=3D"font-weight:bold">Reply-To: </span> &lt;<a =
href=3D"mailto:user@hadoop.apache.org" target=3D"_blank">user@hadoop.apache=
.org</a>&gt;<br><span style=3D"font-weight:bold">Date: </span> Monday, Nove=
mber 2, 2015 at 3:56 PM<br><span style=3D"font-weight:bold">To: </span> &lt=
;<a href=3D"mailto:user@hadoop.apache.org" target=3D"_blank">user@hadoop.ap=
ache.org</a>&gt;<br><span style=3D"font-weight:bold">Subject: </span> Max P=
arallel task executors<br></div><div><div class=3D"h5"><div><br></div><div =
dir=3D"ltr"><div><div><div><div>Hi Team,<br><br></div>I&#39;ve a cloudera c=
luster of 4 nodes. Whenever i submit a job my only 31 parallel tasks are ex=
ecuted whereas my machines have more CPU available but still YARN/AM does n=
ot create more task. <br><br></div>Is there any configuration which I can c=
hange to start more MAP/REDUCER task in parallel?<br><br></div><div>Each ma=
chine in my cluster has 24 CPUs.<br></div><div><br></div>Regards,<br></div>=
Sandeep<br></div></div></div></span></div>
</blockquote></div><br></div>

--001a11c12fd66f202c0523dde943--