Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0AE11102B5 for ; Wed, 9 Jul 2014 14:02:14 +0000 (UTC) Received: (qmail 10736 invoked by uid 500); 9 Jul 2014 14:02:08 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 10626 invoked by uid 500); 9 Jul 2014 14:02:08 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 10613 invoked by uid 99); 9 Jul 2014 14:02:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jul 2014 14:02:07 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of kawa.adam@gmail.com designates 209.85.223.173 as permitted sender) Received: from [209.85.223.173] (HELO mail-ie0-f173.google.com) (209.85.223.173) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jul 2014 14:02:03 +0000 Received: by mail-ie0-f173.google.com with SMTP id x19so3537161ier.32 for ; Wed, 09 Jul 2014 07:01:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=4zJtUSLD0M+UXYaFhZ4qYSNXvpoJdqGdRcfQaLAwaTc=; b=uWQLqqCs/KYnfYo1c6IdiiST1OeHH1WNNfai73wajOxejTkP1IiYuDKvql2oa+F6gi LPfJvbTHV6XnckOWG887GwzahbkuozbOEKaWfnRa4ph751Au6UUOar1CxitGhJ7+rw69 xClzgzM7v33wILfwvINANQWWj00V/Ueyt6Fe5Fg3una6YwPNqqmEoiAGLJUUdVdPUKa7 bF+Sx/bhPEh4RSB9XyOgeNzJ66smSYZ5VuBk5YwbbkVnxqcOnnJnqiJ6R2SxMVSjTtN1 MQC8ZF0H6mn78+P26zAGEKsEETBAXUT9/Te9t3qhdzEH5HI/jBJTA8zvq05YJ1eHd6Uw ZRMA== MIME-Version: 1.0 X-Received: by 10.42.25.19 with SMTP id y19mr47248885icb.45.1404914502709; Wed, 09 Jul 2014 07:01:42 -0700 (PDT) Received: by 10.42.24.69 with HTTP; Wed, 9 Jul 2014 07:01:42 -0700 (PDT) In-Reply-To: References: Date: Wed, 9 Jul 2014 16:01:42 +0200 Message-ID: Subject: Re: The number of simultaneous map tasks is unexpected. From: Adam Kawa To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf3042745adfb73504fdc32584 X-Virus-Checked: Checked by ClamAV on apache.org --20cf3042745adfb73504fdc32584 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Tomek, You have 9.26GB in 4 nodes what is 2.315GB on average. What is your value of yarn.nodemanager.resource.memory-mb? You consume 1GB of RAM per container (8 containers running =3D 8GB of memor= y used). My idea is that, after running 8 containers (1 AM + 7 map tasks), you have only 315MB of available memory on each NodeManager. Therefore, when you request 1GB to get a container for #8 map task, there is no NodeManager than can give you a whole 1GB (despite having more than 1GB of aggregated memory on the cluster). To verify this, please check the value of yarn.nodemanager.resource.memory-mb. Thanks, Adam PS1. Just our of curiosity. What are your values of *yarn.nodemanager.resource.cpu-vcores* (is not it 2?) *yarn.resourcemanager.scheduler.class* (I assume that Fair Scheduler, but just to confirm. Could you have any non-default settings in your scheduler's configuration that limit the number of resources per user?) *yarn.nodemanager.linux-container-executor.resources-handler.class* ? PS2. "I am comparing M/R implementation with a custom one, where one node is dedicated for coordination and I utilize 4 slaves fully for computation." Note that this might not work on a larger scale, because "one node is dedicated for coordination" might become the bottleneck. This is one of a couple of reasons why YARN and original MapReduce at Google have decided to run coordination processes on slave nodes. 2014-07-09 9:47 GMT+02:00 Tomasz Guzia=C5=82ek : > Thank you for your assistance, Adam. > > Containers running | Memory used | Memory total | Memory reserved > 8 | 8 GB | 9.26 GB > | 0 B > > Seems like you are right: the ApplicationMaster is occupying one slot as = I > have 8 containers running, but 7 map tasks. > > Again, I revised my information about m1.large instance on EC2. There are > only 2 cores available per node giving 4 computing units (ECU units > introduced by Amazon). So 8 slots at a time is expected. However, > scheduling AM on a slave node ruins my experiment. I am comparing M/R > implementation with a custom one, where one node is dedicated for > coordination and I utilize 4 slaves fully for computation. This one core > for AM is extending the execution time by a factor of 2. Does any one hav= e > an idea how to have 8 map tasks running? > > Pozdrawiam / Regards / Med venlig hilsen > Tomasz Guzia=C5=82ek > > > 2014-07-09 0:56 GMT+02:00 Adam Kawa : > > If you run an application (e.g. MapReduce job) on YARN cluster, first the >> Application Master will be is started on some slave node to coordinate t= he >> execution of all tasks within the job. The ApplicationMaster and tasks t= hat >> belong to its application run in the containers controlled by the >> NodeManagers. >> >> Maybe, you simply run 8 containers on your YARN cluster and 1 container >> is consumed by MapReduce AppMaster and 7 containers are consumed by map >> tasks. But it seems not to be a root cause of you problem, because >> according to your settings you should be able to run 16 containers >> maximally. >> >> Another idea might be that your are bottlenecked by the amount of memory >> on the cluster (each container consumes memory) and despite having vcore= (s) >> available, you can not launch new tasks. When you go to the ResourceMana= ger >> Web UI, do you see that you utilize whole cluster memory? >> >> >> >> 2014-07-08 21:06 GMT+02:00 Tomasz Guzia=C5=82ek : >> >> I was not precise when describing my cluster. I have 4 slave nodes and a >>> separate master node. The master has ResourceManager role (along with >>> JobHistory role) and the rest have NodeManager roles. If this really is= an >>> ApplicationMaster, is it possible to schedule it on the master node? Th= is >>> single waiting map task is doubling my execution time. >>> >>> Pozdrawiam / Regards / Med venlig hilsen >>> Tomasz Guzia=C5=82ek >>> >>> >>> 2014-07-08 18:42 GMT+02:00 Adam Kawa : >>> >>> Is not your MapReduce AppMaster occupying one slot? >>>> >>>> Sent from my iPhone >>>> >>>> > On 8 jul 2014, at 13:01, Tomasz Guzia=C5=82ek >>>> wrote: >>>> > >>>> > Hello all, >>>> > >>>> > I am running a 4-nodes CDH5 cluster on Amazon EC2 . The instances >>>> used are m1.large, so I have 4 cores (2 core x 2 unit) per node. My HB= ase >>>> table has 8 regions, so I expected at least 8 (if not 16) mapper tasks= to >>>> run simultaneously. However, only 7 are running and 1 is waiting for a= n >>>> empty slot. Why this surprising number came up? I have checked that th= e >>>> regions are equally distributed on the region servers (2 per node). >>>> > >>>> > My properties in the job: >>>> > Configuration mapReduceConfiguration =3D HBaseConfiguration.create()= ; >>>> > mapReduceConfiguration.set("hbase.client.max.perregion.tasks", "4"); >>>> > mapReduceConfiguration.set("mapreduce.tasktracker.map.tasks.maximum"= , >>>> "16"); >>>> > >>>> > My properties in the CDH: >>>> > yarn.scheduler.minimum-allocation-vcores =3D 1 >>>> > yarn.scheduler.maximum-allocation-vcores =3D 4 >>>> > >>>> > Do I miss some property? Please share your experience. >>>> > >>>> > Best regards >>>> > Tomasz >>>> >>> >>> >> > --20cf3042745adfb73504fdc32584 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Tomek,

You have=C2=A09.26GB in 4 nodes w= hat is=C2=A02.315GB on average. What is your value of yarn.nodemanag= er.resource.memory-mb?

You consume 1GB of RAM per container (8 containers runn= ing =3D 8GB of memory used). My idea is that, after running 8 containers (1= AM + 7 map tasks), you have only 315MB of available memory on each NodeMan= ager. Therefore, when you request 1GB to get a container for #8 map task, t= here is no NodeManager than can give you a whole 1GB (despite having more t= han 1GB of aggregated memory on the cluster).

To verify this, please check the value of yarn.nodemana= ger.resource.memory-mb.

Thanks,
Ada= m

PS1.=C2=A0
Just our of curiosity= . What are your values of=C2=A0
yarn.nodemanager.resource.cpu-vcores (is not it 2?)<= /div>
yarn.resourcemanager.scheduler.class (I assume that Fair S= cheduler, but just to confirm. Could you have any non-default settings in y= our scheduler's configuration that limit the number of resources per us= er?)
yarn.nodemanager.linux-container-executor.resources-hand= ler.class
?

PS2.
"I am comparing M= /R implementation with a custom one, where one node is dedicated for coordi= nation and I utilize 4 slaves fully for computation."

Not= e that this might not work on a larger scale, because "one node is dedicated f= or coordination" might become the bottleneck. This is one of a couple = of reasons why YARN and original MapReduce at Google have decided to run co= ordination processes on slave nodes.



2014-07-09 9:47 GMT+02:00 Tomasz Guzia=C5=82ek= <tomasz@guzialek.info>:
Thank y= ou for your assistance, Adam.

Containers running | Memory= used | Memory total | Memory reserved
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 8 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 8 GB |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 9.26 GB |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 B

Seems like you are right: the ApplicationMaster is occupying one = slot as I have 8 containers running, but 7 map tasks.

Again, = I revised my information about m1.large instance on EC2. There are only 2 c= ores available per node giving 4 computing units (ECU units introduced by A= mazon). So 8 slots at a time is expected. However, scheduling AM on a slave= node ruins my experiment. I am comparing M/R implementation with a custom = one, where one node is dedicated for coordination and I utilize 4 slaves fu= lly for computation. This one core for AM is extending the execution time b= y a factor of 2. Does any one have an idea how to have 8 map tasks running?=

Pozdrawiam / Regards / Med venlig hilsen
Tomasz Guzia=C5=82ek


2014-07-09 0:56 GMT+02:00 Adam Kaw= a <kawa.adam@gmail.com>:

If you run an application (e.g. MapReduce job) on YAR= N cluster, first the Application Master will be is started on some slave no= de to coordinate the execution of all tasks within the job. The Application= Master and tasks that belong to its application run in the containers contr= olled by the NodeManagers.

Maybe, you simply run 8 containers on your YARN c= luster and 1 container is consumed by MapReduce AppMaster and 7 containers = are consumed by map tasks. But it seems not to be a root cause of you probl= em, because according to your settings you should be able to run 16 contain= ers maximally.

Another idea might be that your are bottlenecked by the= amount of memory on the cluster (each container consumes memory) and despi= te having vcore(s) available, you can not launch new tasks. When you go to = the ResourceManager Web UI, do you see that you utilize whole cluster memor= y?



2014-07-08 21:06 GMT+02:00 Tomasz Guzia=C5=82ek &= lt;tomasz@guziale= k.info>:

I was not precise when desc= ribing my cluster. I have 4 slave nodes and a separate master node. The mas= ter has ResourceManager role (along with JobHistory role) and the rest have= NodeManager roles. If this really is an ApplicationMaster, is it possible = to schedule it on the master node? This single waiting map task is doubling= my execution time.

Pozdrawiam / Regards / Me= d venlig hilsen
Tomasz Guzia=C5=82ek


2014-07-08 18:42 GMT+02:00 Adam Kawa <kawa.adam@gmail.com>:

Is not your MapReduce AppMaster occupying one slot?

Sent from my iPhone

> On 8 jul 2014, at 13:01, Tomasz Guzia=C5=82ek <tomaszguzialek@gmail.com> = wrote:
>
> Hello all,
>
> I am running a 4-nodes CDH5 cluster on Amazon EC2 . The instances used= are m1.large, so I have 4 cores (2 core x 2 unit) per node. My HBase table= has 8 regions, so I expected at least 8 (if not 16) mapper tasks to run si= multaneously. However, only 7 are running and 1 is waiting for an empty slo= t. Why this surprising number came up? I have checked that the regions are = equally distributed on the region servers (2 per node).
>
> My properties in the job:
> Configuration mapReduceConfiguration =3D HBaseConfiguration.create();<= br> > mapReduceConfiguration.set("hbase.client.max.perregion.tasks"= ;, "4");
> mapReduceConfiguration.set("mapreduce.tasktracker.map.tasks.maxim= um", "16");
>
> My properties in the CDH:
> yarn.scheduler.minimum-allocation-vcores =3D 1
> yarn.scheduler.maximum-allocation-vcores =3D 4
>
> Do I miss some property? Please share your experience.
>
> Best regards
> Tomasz




--20cf3042745adfb73504fdc32584--