Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6D458D23C for ; Thu, 8 Nov 2012 23:20:32 +0000 (UTC) Received: (qmail 93765 invoked by uid 500); 8 Nov 2012 23:20:27 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 93488 invoked by uid 500); 8 Nov 2012 23:20:27 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 93481 invoked by uid 99); 8 Nov 2012 23:20:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Nov 2012 23:20:27 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of msgdh8@gmail.com designates 209.85.212.48 as permitted sender) Received: from [209.85.212.48] (HELO mail-vb0-f48.google.com) (209.85.212.48) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Nov 2012 23:20:20 +0000 Received: by mail-vb0-f48.google.com with SMTP id e21so3898347vbm.35 for ; Thu, 08 Nov 2012 15:19:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=tfgvbqJ4D9jA7mRjTqLlfGwXw8EG8RZvgjJeb5OPNd4=; b=ez201/uZcDErC55aTFTigoyoRwtPXGfpl9WIoRCXpLSkDhlImUAsYq9rsju3ySODGw 8ZxJZxXeRBliVpCZwOH2OvpEN/nFNs5RxcfPmsP2DRFEewU8/RDqKk2daj7uMMPP7Asz rY/laPnOwWDbuiDicW55qCU7mWeuQmnJv7lvYNsVtstmLRdrZr+Ug2tRMbyYcRS3UDtQ O6tkRSELE/WdmQjzhDWKolcQKQeZBkls0tPZwCKgE06DaKj62y8E8OuQ1LqVGx0VnAb/ c6yUJEgt+MyVpOzXgx16SAHTNVMUGepPdJkmhKKN9wkJ1DhbYaz6HJmZG8uUzTom5XdB DOCQ== MIME-Version: 1.0 Received: by 10.220.238.148 with SMTP id ks20mr8793225vcb.5.1352416799465; Thu, 08 Nov 2012 15:19:59 -0800 (PST) Sender: msgdh8@gmail.com Received: by 10.58.15.137 with HTTP; Thu, 8 Nov 2012 15:19:59 -0800 (PST) In-Reply-To: References: Date: Thu, 8 Nov 2012 17:19:59 -0600 X-Google-Sender-Auth: 7mvsELluz6iGikIcGrh9_W6so7c Message-ID: Subject: Re: Fair Scheduler question: Fair share and its effect on max capacity From: Matt Goeke To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=14dae9d253b4eba23b04ce0412c7 X-Virus-Checked: Checked by ClamAV on apache.org --14dae9d253b4eba23b04ce0412c7 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Looks like my phrasing was off :) When I said it is never able to hit max capacity I meant max capacity for the pool (e.g. we never saw it take up the full 200 maps AND even if every jobs uses 1 mapper could never get to 200 concurrent jobs for that pool). -- Matt On Thu, Nov 8, 2012 at 5:18 PM, Nan Zhu wrote: > You set maxMaps to 200, > > so the maximum running mappers should be no more than 200 > > Best, > > -- > Nan Zhu > School of Computer Science, > McGill University > > > On Thursday, 8 November, 2012 at 6:12 PM, Matt Goeke wrote: > > Pretty straight forward question but can the fair share factor actually > impact the total number of jobs / slots a pool can take up even if it is > the only pool with active jobs submitted? > > We currently have a pool that has this configuration: > "minMaps": 2, > > "minReduces": 1, > "maxMaps": 200, > "maxReduces": 66, > "maxRunningJobs": 200, > "minSharePreemptionTimeout": 300, > "weight": "4.0" > > The total cluster capacity is over above 250 mappers but we are finding t= hat this pool is never able to hit that max capacity for maps OR jobs even = during load tests. I was about to bump the minMaps property but I wanted to= confirm that this could potentially help alleviate our issue. > > -- > Matt > > > --14dae9d253b4eba23b04ce0412c7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Looks like my phrasing was off :)

When I said it is neve= r able to hit max capacity I meant max capacity for the pool (e.g. we never= saw it take up the full 200 maps AND even if every jobs uses 1 mapper coul= d never get to 200 concurrent jobs for that pool).

--
Matt

<= br>
On Thu, Nov 8, 2012 at 5:18 PM, Nan Zhu <= zhunansjtu@gmail.com> wrote:
You set maxMaps to 200,=A0

so the maximum running mappers sh= ould be no more than 200

Best,

--=A0
Nan Zhu
= School of Computer Science,
McGill University


=20

On Thursday, 8 November, 2012 at= 6:12 PM, Matt Goeke wrote:

Pretty straight forward question but ca= n the fair share factor actually impact the total number of jobs / slots a = pool can take up even if it is the only pool with active jobs submitted?
We currently have a pool that has this configuration:
"minMaps": 2,


"=
minReduces":<=
/span> 1,



"maxMaps": 200,



"maxReduces": 66,



"maxRunningJobs": 200,



"minSharePreemptionTimeout": 300,



"weight": "4.0"







=
The total cluster capacity is over above 2=
50 mappers but we are finding that this pool is never able to hit that max =
capacity for maps OR jobs even during load tests. I was about to bump the minMaps prope=
rty but I wanted to confirm that this could potentially help=A0alleviate=A0=
our issue.






--<=
/span>



Matt
=20 =20 =20 =20
=20


--14dae9d253b4eba23b04ce0412c7--