Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D579718524 for ; Mon, 9 Nov 2015 13:59:08 +0000 (UTC) Received: (qmail 26864 invoked by uid 500); 9 Nov 2015 13:59:04 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 26736 invoked by uid 500); 9 Nov 2015 13:59:03 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 26721 invoked by uid 99); 9 Nov 2015 13:59:03 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Nov 2015 13:59:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 34A6E1A09EE for ; Mon, 9 Nov 2015 13:59:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.98 X-Spam-Level: ** X-Spam-Status: No, score=2.98 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, T_RP_MATCHES_RCVD=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id qR69XNn0JtLi for ; Mon, 9 Nov 2015 13:58:57 +0000 (UTC) Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [119.145.14.66]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 0FE8F2303F for ; Mon, 9 Nov 2015 13:58:48 +0000 (UTC) Received: from 172.24.1.51 (EHLO szxeml427-hub.china.huawei.com) ([172.24.1.51]) by szxrg03-dlp.huawei.com (MOS 4.4.3-GA FastPath queued) with ESMTP id BQR09398; Mon, 09 Nov 2015 20:18:18 +0800 (CST) Received: from SZXEML510-MBX.china.huawei.com ([169.254.3.4]) by szxeml427-hub.china.huawei.com ([10.82.67.182]) with mapi id 14.03.0235.001; Mon, 9 Nov 2015 20:18:10 +0800 From: Brahma Reddy Battula To: "user@hadoop.apache.org" Subject: RE: Max Parallel task executors Thread-Topic: Max Parallel task executors Thread-Index: AQHRFVjy/iBHdh/HjUqtoAkUTjC+CJ6Jzs0AgASUSwCAAAkagIAAAJSAgAAEcACAAAGngIAAAs6AgACLBTSAA5h5gIAAAZEAgAAjAACAAOhjkA== Date: Mon, 9 Nov 2015 12:18:10 +0000 Message-ID: <8AD4EE147886274A8B495D6AF407DF698E4A02B4@szxeml510-mbx.china.huawei.com> References: <8AD4EE147886274A8B495D6AF407DF698E49F165@szxeml510-mbx.china.huawei.com> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.18.240.230] Content-Type: multipart/alternative; boundary="_000_8AD4EE147886274A8B495D6AF407DF698E4A02B4szxeml510mbxchi_" MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020206.5640A68C.023B,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0, ip=169.254.3.4, so=2013-05-26 15:14:31, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: b7a34a321b859b98481db34c889bcc20 --_000_8AD4EE147886274A8B495D6AF407DF698E4A02B4szxeml510mbxchi_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I'm glad to hear it helped. Thanks & Regards Brahma Reddy Battula ________________________________ From: sandeep das [yarnhadoop@gmail.com] Sent: Monday, November 09, 2015 11:54 AM To: user@hadoop.apache.org Subject: Re: Max Parallel task executors After increasing yarn.nodemanager.resource.memory-mb to 24 GB more number o= f parallel map tasks are being spawned. Its resolved now. Thanks a lot for your input. Regards, Sandeep On Mon, Nov 9, 2015 at 9:49 AM, sandeep das > wrote: BTW Laxman according to the formula that you had provided it turns out that= only 8 jobs per node will be initiated which is matching with what i'm see= ing on my setup. min (yarn.nodemanager.resource.memory-mb / mapreduce.[map|reduce].memory.mb= , yarn.nodemanager.resource.cpu-vcores / mapreduce.[map|reduce].cpu.vcor= es) yarn.nodemanager.resource.memory-mb: 16 GB mapreduce.map.memory.mb: 2 GB yarn.nodemanager.resource.cpu-vcores: 80 mapreduce.map.cpu.vcores: 1 So if apply the formula then min(16/2, 80/1) -> min(8,80) -> 8 Should i reduce memory per map operation or increase memory for resource ma= nager? On Mon, Nov 9, 2015 at 9:43 AM, sandeep das > wrote: Thanks Brahma and Laxman for your valuable input. Following are the statistics available on YARN RM GUI. Memory Used : 0 GB Memory Total : 64 GB (16*4 =3D 64 GB) VCores Used: 0 VCores Total: 320 (Earlier I had mentioned that I've configured 40 Vcores b= ut recently I increased to 80 that's why its appearing 80*4 =3D 321) Note: These statistics were captured when there was no job running in backg= round. Let me know whether it was sufficient to nail the issue. If more informatio= n is required please let me know. Regards, Sandeep On Fri, Nov 6, 2015 at 7:04 PM, Brahma Reddy Battula > wrote: The formula for determining the number of concurrently running tasks per no= de is: min (yarn.nodemanager.resource.memory-mb / mapreduce.[map|reduce].memory.mb= , yarn.nodemanager.resource.cpu-vcores / mapreduce.[map|reduce].cpu.vcor= es) . For you scenario : As you told yarn.nodemanager.resource.memory-mb is configured to 16 GB and = yarn.nodemanager.resource.cpu-vcores configured to 40. and I am thinking mapreduce.map/reduce.memory.mb, mapreduce.map/reduce.cpu.vcores default val= ues. min (16GB/1GB,40Core/1Core )=3D16 tasks for Node. Then total should be 16*4= =3D64 (63+1AM).. I am thinking, Two Nodemanger's are unhealthy (OR) you might have configure= d mapreduce.map/reduce.memory.mb=3D2GB(or 5 core). As laxman pointed you can post RMUI or you can cross check like above. Hope this helps. Thanks & Regards Brahma Reddy Battula ________________________________ From: Laxman Ch [laxman.lux@gmail.com] Sent: Friday, November 06, 2015 6:31 PM To: user@hadoop.apache.org Subject: Re: Max Parallel task executors Can you please copy paste the cluster metrics from RM dashboard. Its under http://rmhost:port/cluster/cluster In this page, check under Memory Total vs Memory Used and VCores Total vs V= Cores Used On 6 November 2015 at 18:21, sandeep das > wrote: HI Laxman, Thanks for your response. I had already configured a very high value for ya= rn.nodemanager.resource.cpu-vcores e.g. 40 but still its not increasing mor= e number of parallel tasks to execute but if this value is reduced then it = runs less number of parallel tasks. As of now yarn.nodemanager.resource.memory-mb is configured to 16 GB and ya= rn.nodemanager.resource.cpu-vcores configured to 40. Still its not spawning more tasks than 31. Let me know if more information is required to debug it. I believe there is= upper limit after which yarn stops spawning tasks. I may be wrong here. Regards, Sandeep On Fri, Nov 6, 2015 at 6:15 PM, Laxman Ch > wrote: Hi Sandeep, Please configure the following items to the cores and memory per node you w= anted to allocate for Yarn containers. Their defaults are 8 cores and 8GB. So that's the reason you were stuck at = 31 (4nodes * 8cores - 1 AppMaster) http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-de= fault.xml yarn.nodemanager.resource.cpu-vcores yarn.nodemanager.resource.memory-mb On 6 November 2015 at 17:59, sandeep das > wrote: May be to naive to ask but How do I check that? Sometimes there are almost 200 map tasks pending to run but at a time only = 31 runs. On Fri, Nov 6, 2015 at 5:57 PM, Chris Mawata > wrote: Also check that you have more than 31 blocks to process. On Nov 6, 2015 6:54 AM, "sandeep das" > wrote: Hi Varun, I tried to increase this parameter but it did not increase number of parall= el tasks but if It is decreased then YARN reduces number of parallel tasks.= I'm bit puzzled why its not increasing more than 31 tasks even after its v= alue is increased. Is there any other configuration as well which controls on how many maximum= tasks can execute in parallel? Regards, Sandeep On Tue, Nov 3, 2015 at 7:29 PM, Varun Vasudev > wrote: The number of parallel tasks that are run depends on the amount of memory a= nd vcores on your machines and the amount of memory and vcores required by = your mappers and reducers. The amount of memory can be set via yarn.nodeman= ager.resource.memory-mb(the default is 8G). The amount of vcores can be set= via yarn.nodemanager.resource.cpu-vcores(the default is 8 vcores). -Varun From: sandeep das > Reply-To: > Date: Monday, November 2, 2015 at 3:56 PM To: > Subject: Max Parallel task executors Hi Team, I've a cloudera cluster of 4 nodes. Whenever i submit a job my only 31 para= llel tasks are executed whereas my machines have more CPU available but sti= ll YARN/AM does not create more task. Is there any configuration which I can change to start more MAP/REDUCER tas= k in parallel? Each machine in my cluster has 24 CPUs. Regards, Sandeep -- Thanks, Laxman -- Thanks, Laxman --_000_8AD4EE147886274A8B495D6AF407DF698E4A02B4szxeml510mbxchi_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

I'm glad to hear it helped.

Thanks & Regards

 Brahma Reddy Battul= a

 


From: sandeep das [yarnhadoop@gmail.com]<= br> Sent: Monday, November 09, 2015 11:54 AM
To: user@hadoop.apache.org
Subject: Re: Max Parallel task executors

After increasing yarn.nodemanager.resource.memory-mb to 24 GB more number of parallel map tasks are being spawned. Its resolved= now.
Thanks a lot for your input.

Regards,
Sandeep

On Mon, Nov 9, 2015 at 9:49 AM, sandeep das <yarnhadoop@gm= ail.com> wrote:
BTW Laxman according to the formula that you had provided it turns out= that only 8 jobs per node will be initiated which is matching with what i'= m seeing on my setup.

min (yarn.nodemanager.resource.memory-mb <= /font>/ mapreduce.[map|reduce].memory.mb,
     yarn.nodemanager.resourc= e.cpu-vcores / mapreduce.[map|reduce].cpu.vcores)


yarn.n= odemanager.resource.memory-mb: 16 GB

mapreduce.map= .memory.mb: 2 GB

yarn.nodemanager.resourc= e.cpu-vcores: 80

mapreduce.map= .cpu.vcores: 1

So if apply the formula then min(16/2, 80/1) -> min(8,80) -> 8

Should i reduce memory per map operation or increase memory for resource manager?

On Mon, Nov 9, 2015 at 9:43 AM, sandeep das <yarnhadoop@gm= ail.com> wrote:
Thanks Brahma and Laxman for your valuable input.

Following are the statistics available on YARN RM GUI.

Memory Used : 0 GB
Memory Total : 64 GB (16*4 =3D 64 GB)
VCores Used: 0
VCores Total: 320 (Earlier I had mentioned that I've configured 40 Vcores b= ut recently I increased to 80 that's why its appearing 80*4 =3D 321)

Note: These statistics were captured when there was no job running in = background.

Let me know whether it was sufficient to nail the issue. If more informatio= n is required please let me know.

Regards,
Sandeep


On Fri, Nov 6, 2015 at 7:04 PM, Brahma Reddy Bat= tula <bra= hmareddy.battula@huawei.com> wrote:

The formula for determining the number of concurrent= ly running tasks per node is:

min (yarn.no= demanager.resource.memory-mb / mapreduce.[map|reduce].memory.mb= ,
     yarn.nodemanager.resourc= e.cpu-vcores / mapreduce.[map|reduce].cpu.vcores) .


For you scenario :

As you told yarn.nodemanager.resource.memory-mb is configured to 16 GB= and yarn.nodemanager.resource.cpu-vcores configured to 40. and I am thinking
mapreduce.map/reduce.memory.mb, mapreduce.map/reduce.cpu.vcores default values.

min (16GB/1GB,40Core/1Core )=3D16 tasks for Node. Then to= tal should be 16*4=3D64  (63+1AM)..

I am thinking, Two Nodemanger's are unhealthy (OR) = = you might have configured mapreduce.map/reduce.memory.mb=3D2GB(or 5 core).

As laxman pointed you can post RMUI or you can cross check= like above.

Hope this helps.



Thanks & Regards

 Brahma Reddy Battul= a

 


From: Laxman Ch [laxman.lux@gmail.com]
Sent: Friday, November 06, 2015 6:31 PM
To: user= @hadoop.apache.org
Subject: Re: Max Parallel task executors

Can you please copy paste the cluster metrics from RM dash= board.
Its under http://rmhost:port/cluster/cluster

In this page, check under Memory Total vs Memory Used and VCores Total= vs VCores Used

On 6 November 2015 at 18:21, sandeep das <yarnhadoop@gm= ail.com> wrote:

On Fri, Nov 6, 2015 at 6:15 PM, Laxman Ch <laxman.lux@gm= ail.com> wrote:
Hi Sandeep,

Please configure the following items to the cores and memory per node = you wanted to allocate for Yarn containers.
Their defaults are 8 cores and 8GB. So that's the reason you were stuc= k at 31 (4nodes * 8cores - 1 AppMaster)

On 6 November 2015 at 17:59, sandeep das <yarnhadoop@gm= ail.com> wrote:
May be to naive to ask but How do I check that? 
Sometimes there are almost 200 map tasks pending to run but at a time only = 31 runs.

On Fri, Nov 6, 2015 at 5:57 PM, Chris Mawata <chris.mawat= a@gmail.com> wrote:

Also check that you have more than 31 blocks to process.

On Nov 6, 2015 6:54 AM, "sandeep das" = <yarnhadoop@gm= ail.com> wrote:
Hi Varun,

I tried to increase this parameter but it did not increase number of p= arallel tasks but if It is decreased then YARN reduces number of parallel t= asks. I'm bit puzzled why its not increasing more than 31 tasks even after = its value is increased.

Is there any other configuration as well which controls on how many ma= ximum tasks can execute in parallel?

Regards,
Sandeep

On Tue, Nov 3, 2015 at 7:29 PM, Varun Vasudev <vvasudev@apach= e.org> wrote:
The number of parallel tasks that are run depends on the amount of mem= ory and vcores on your machines and the amount of memory and vcores require= d by your mappers and reducers. The amount of memory can be set via ya= rn.nodemanager.resource.memory-mb(the default is 8G). The amount of vcores can be set via yarn.nodem= anager.resource.cpu-vcores(the default is 8 vcores).

-Varun

From: sandeep das <yarnhadoop@gmail.com> Reply-To: <user@hadoop.apache.org>
Date: Monday, November 2, 2015 at 3= :56 PM
To: <user@hadoop.apache.org>
Subject: Max Parallel task executor= s

Hi Team,

I've a cloudera cluster of 4 nodes. Whenever i submit a job my only 31 para= llel tasks are executed whereas my machines have more CPU available but sti= ll YARN/AM does not create more task.

Is there any configuration which I can change to start more MAP/REDUCER tas= k in parallel?

Each machine in my cluster has 24 CPUs.

Regards,
Sandeep





--
Thanks,
Laxman




--
Thanks,
Laxman



--_000_8AD4EE147886274A8B495D6AF407DF698E4A02B4szxeml510mbxchi_--