Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 116781047A for ; Wed, 15 Oct 2014 13:36:50 +0000 (UTC) Received: (qmail 61141 invoked by uid 500); 15 Oct 2014 13:36:44 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 61015 invoked by uid 500); 15 Oct 2014 13:36:44 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 61005 invoked by uid 99); 15 Oct 2014 13:36:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Oct 2014 13:36:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of shahab.yunus@gmail.com designates 209.85.217.170 as permitted sender) Received: from [209.85.217.170] (HELO mail-lb0-f170.google.com) (209.85.217.170) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Oct 2014 13:36:18 +0000 Received: by mail-lb0-f170.google.com with SMTP id u10so1058875lbd.1 for ; Wed, 15 Oct 2014 06:36:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=PV6JYFOpr26NtdFIHRTv8Pd8WXzfOkqdft/lKd7WfQY=; b=VxTOiqZeUu+cBLKYTjx26zNsn69H42KzZL1TjmFgaiBguaGCm+8mI4LTIsM9T8a9kE t+dt/esZgh5w1FN5BTfJ8lFkxBPuDgzn8plvT86ISkmr4T7ltfgbwPG7JVHUfSz+p+Pq 86hn3uZJ4cRcb00bQoXy4H/Idl77NScPMzDArnsZN64kgFHH9Eb6w2CTpGAiyOo8ReBy EU4FYC16d16A81JWmFk/5lBa7d9M4ddj/iq35/Q69vqJU7P2xRcepYfEeDAYJFL+V5zc +6x3L0+reDFOmRBINK25+pHadRNcBdr3pktVNIyuXqHQ4Dv+8XmrKXQphMCRUssiENdC 1/fw== MIME-Version: 1.0 X-Received: by 10.112.135.42 with SMTP id pp10mr12409780lbb.43.1413380176741; Wed, 15 Oct 2014 06:36:16 -0700 (PDT) Received: by 10.25.33.145 with HTTP; Wed, 15 Oct 2014 06:36:16 -0700 (PDT) In-Reply-To: <543E758C.8080805@datametica.com> References: <543E63FB.9000400@datametica.com> <543E660D.4050503@datametica.com> <543E758C.8080805@datametica.com> Date: Wed, 15 Oct 2014 09:36:16 -0400 Message-ID: Subject: Re: number of mappers allowed in a container in hadoop2 From: Shahab Yunus To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=089e01175e795e091c05057637fb X-Virus-Checked: Checked by ClamAV on apache.org --089e01175e795e091c05057637fb Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Explanation here. http://stackoverflow.com/questions/24070557/what-is-the-relation-between-ma= preduce-map-memory-mb-and-mapred-map-child-jav https://support.pivotal.io/hc/en-us/articles/201462036-Mapreduce-YARN-Memor= y-Parameters http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-common/Clus= terSetup.html (scroll towards the end.) Regards, Shahab On Wed, Oct 15, 2014 at 9:24 AM, SACHINGUPTA wrote: > I have one more doubt i was reading this > > > http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_m= anually_book/content/rpm-chap1-11.html > > there is one property as > > mapreduce.map.memory.mb =3D 2*1024 MB > mapreduce.reduce.memory.mb =3D 2 * 2 =3D 4*1024 MB > what are these properties mapreduce.map.memory.mb and > mapreduce.reduce.memory.mb > > On Wednesday 15 October 2014 06:17 PM, Shahab Yunus wrote: > > It cannot run more mappers (tasks) in parallel than the underlying cores > available. Just like it cannot run multiple mappers in parallel if each > mapper's (task's) memory requirements are greater than allocated and > available container size configured on each node. > > The links that I provided earlier...see the following section in that > one: > Section:"Configuring YARN" > > Also this: > http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-= consuming-gotchas/ > Section "1. YARN Concurrency (aka =E2=80=9CWhat Happened to Slots?=E2=80= =9D)" > > This should help in putting things in perspective regarding how resource > allocation for each task, container and resources available on the node > relate to each other. > > Regards, > Shahab > > On Wed, Oct 15, 2014 at 8:18 AM, SACHINGUPTA > wrote: > >> but Shahab if i have only 4 core machine then how yarn can run more the= n >> 4 mappers in parallel >> On Wednesday 15 October 2014 05:45 PM, Shahab Yunus wrote: >> >> It depends on memory settings as well, that how much you want to assign >> resources to each container. Then yarn will run as many mappers in paral= lel >> as possible. >> >> See this: >> http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/ >> >> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_= manually_book/content/rpm-chap1-11.html >> >> Regards, >> Shahab >> >> On Wed, Oct 15, 2014 at 8:09 AM, SACHINGUPTA >> wrote: >> >>> Hi guys >>> >>> I have situation in which i have machine with 4 processor and i have 5 >>> containers so does it mean i can have only 4 mappers running parallely = at a >>> time >>> >>> and number of mappers is not dependent on the number of containers in a >>> machine then what is the use of container concept >>> >>> sorry if i have asked anything obvious. >>> >>> -- >>> Thanks >>> Sachin Gupta >>> >>> >> >> -- >> Thanks >> Sachin Gupta >> >> > > -- > Thanks > Sachin Gupta > > --089e01175e795e091c05057637fb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On Wed, Oct 15, 2014 a= t 9:24 AM, SACHINGUPTA <sachin@datametica.com> wrote:
=20 =20 =20
I have one more doubt i was reading this

http= ://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manuall= y_book/content/rpm-chap1-11.html

there is one property as

mapreduce.map.memory.mb =3D 2*1024 MB
mapreduce.reduce.memory.mb=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D 2 * 2 =3D 4*1024 MB

what are these properties mapreduce.map.memory.mb and mapreduce.reduce.memory.mb

On Wednesday 15 October 2014 06:17 PM, Shahab Yunus wrote:
It cannot run more mappers (tasks) in parallel than the underlying cores available. Just like it cannot run multiple mappers in parallel if each mapper's (task's) memory requirements are greater than allocated and available container size configured on each node.

The links that I provided earlier...see the following section in that one:
Section:"Configuring YARN"

Section "1. YARN Concurrency (aka =E2=80=9CWhat Happe= ned to Slots?=E2=80=9D)"

This should help in putting things in perspective regarding how resource allocation for each task, container and resources available on the node relate to each other.

Regards,
Shahab

On Wed, Oct 15, 2014 at 8:18 AM, SACHINGUPTA <sachin@datametica.com> wrote:
but Shahab if i have only 4 core machine then how yarn can run more then 4 mappers in parallel
On Wednesday 15 October 2014 05:45 PM, Shahab Yunus wrote:
It depends on memory settings as well, that how much you want to assign resources to each container. Then yarn will run as many mappers in parallel as possible.

See this:

Regards,
Shahab

On Wed, Oct 15, 2014 at 8:09 AM, SACHINGUPTA <sachin@datametica.com> wrote:
Hi guys

I have situation in which i have machine with 4 processor and i have 5 containers so does it mean i can have only 4 mappers running parallely at a time

and number of mappers is not dependent on the number of containers in a machine then what is the use of container concept

sorry if i have asked anything obvious.

--
Thanks
Sachin Gupta


= 

--=20
Thanks
Sachin Gupta


--=20
Thanks
Sachin Gupta

--089e01175e795e091c05057637fb--