Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CB23F1034F for ; Fri, 20 Sep 2013 19:21:13 +0000 (UTC) Received: (qmail 57010 invoked by uid 500); 20 Sep 2013 19:21:08 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 56783 invoked by uid 500); 20 Sep 2013 19:21:06 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 56776 invoked by uid 99); 20 Sep 2013 19:21:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Sep 2013 19:21:05 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ojoshi@hortonworks.com designates 209.85.215.43 as permitted sender) Received: from [209.85.215.43] (HELO mail-la0-f43.google.com) (209.85.215.43) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Sep 2013 19:20:59 +0000 Received: by mail-la0-f43.google.com with SMTP id ep20so693205lab.16 for ; Fri, 20 Sep 2013 12:20:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=sRFyedPCxukINU1CViLpE4RRVbrNUiZDbJsDDkuhegU=; b=AlUd79v85jQex2DK9ULOVHxeaPalS0yo7JOM7dI+uU+VqPiq8U3nX+Qn5ZuW8O9HZR M70yYMk7wCBB5P+l67oYnf0lXOR1DdCM4JXOk6ARg9e2XRUh4ET0OxfLSwOH99prjw1e 6uDXmaMFJvVzf8UcmuAso1VcKFJGAZWOngxK3B3khC5awZSjEaOhHsRv3Okgm8Ho0PJF bVh5Q3uPDULSV/r9e+aoJVC4sx6mNZyZ4drWJ1i9CZK/hKKwWLy78Ecx1URIfkAoAODt ecQsjJyuRmlms4Y9HanhET2kk4UN3eMxIX0FTmiHj74qrwqOP916mWXAocrPb52V8G9G RE9g== X-Gm-Message-State: ALoCoQlYMojscQhhhdhgAl7h8xu9o0O7sTu6vAz1eAw0GYJ1GiOg1EFbNy0+OMUWJJiZPSjZsWp2zlrI3G4ZTMm0LxNIr5sB9hN9g5ZmW2lM9N/6ktyjZ+I= MIME-Version: 1.0 X-Received: by 10.112.146.33 with SMTP id sz1mr7429325lbb.14.1379704839053; Fri, 20 Sep 2013 12:20:39 -0700 (PDT) Received: by 10.112.182.97 with HTTP; Fri, 20 Sep 2013 12:20:38 -0700 (PDT) In-Reply-To: <3129260.Jz0lzkYlD7@p854801> References: <5cd8d054fde44601ab5f0a709ac99f6e@BY2PR09MB062.namprd09.prod.outlook.com> <3129260.Jz0lzkYlD7@p854801> Date: Fri, 20 Sep 2013 12:20:38 -0700 Message-ID: Subject: Re: How to make hadoop use all nodes? From: Omkar Joshi To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b3441d0d3b26b04e6d59071 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b3441d0d3b26b04e6d59071 Content-Type: text/plain; charset=US-ASCII Hi, few more questions (which has 40 containers slots.) >> for total cluster? Please give below details for cluster 1) yarn-site.xml -> what is the resource memory configured for per node? 2) yarn-site.xml -> what is the minimum resource allocation for the cluster? 3) yarn-resource-manager-log (while starting resource manager "export YARN_ROOT_LOGGER=DEBUG,RFA").. I am looking for debug logs.. 4) On RM UI how much total cluster memory is reported (how many total nodes). ( RM UI click on Cluster) 5) which scheduler you are using? Capacity/Fair/FIFO 6) have you configured any user limits/ queue capacity? (please add details). 7) All requests you are making at same priority or with different priorities? (Ideally it will not matter but want to know). Please let us know all the above details. Thanks. Thanks, Omkar Joshi *Hortonworks Inc.* On Fri, Sep 20, 2013 at 6:55 AM, Antoine Vandecreme < antoine.vandecreme@nist.gov> wrote: > Hello Omkar, > > Thanks for your reply. > > Yes, all 4 points are corrects. > However, my application is requesting let say 100 containers on my cluster > which has 40 containers slots. > So I expected to see all containers slots used but that is not the case. > > Just in case it matters, it is the only application running on the server. > > Thanks, > Antoine Vandecreme > > On Thursday, September 19, 2013 04:49:36 PM Omkar Joshi wrote: > > Hi, > > > > Let me clarify few things. > > 1) you are making container requests which are not explicitly looking for > > certain nodes. (No white listing). > > 2) All nodes are identical in terms of resources (memory/cores) and every > > container requires same amount of resources. > > 3) All nodes have capacity to run say 2 containers. > > 4) You have 20 nodes. > > > > Now if an application is running and is requesting 20 containers then you > > can not say that you will get all on different nodes (uniformly > > distributed). It more depends on which node heartbeated to the Resource > > manager at what time and how much memory is available with it and also > how > > many applications are present in queue and how much they are requesting > at > > what request priorities. If it has say sufficient memory to run 2 > > containers then they will get allocated (This allocation is quite complex > > ..I am assuming very simple "*" reuqest). So you may see few running 2, > few > > running 1 where as few with 0 containers. > > > > I hope it clarifies your doubt. > > > > Thanks, > > Omkar Joshi > > *Hortonworks Inc.* > > > > > > On Thu, Sep 19, 2013 at 7:19 AM, Vandecreme, Antoine < > > > > antoine.vandecreme@nist.gov> wrote: > > > Hi all, > > > > > > I am working with Hadoop 2.0.5 (I plan to migrate to 2.1.0 soon). > > > When I am starting a Job, I notice that some nodes are not used or > > > partially used. > > > > > > For example, if my nodes can hold 2 containers, I notice that some > nodes > > > are not running any or just 1 while others are running 2. > > > All my nodes are configured the same way. > > > > > > Is this an expected behavior (maybe in case others jobs are started) ? > > > Is there a configuration to change this behavior? > > > > > > Thanks, > > > Antoine > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --047d7b3441d0d3b26b04e6d59071 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,

few more questions=A0
(which has 40 containers slots.) >> for total cluster? P= lease give below details

for cluster
1) yarn-site.xml= -> what is the resource memory configured for per node?
2) yarn-site.xml -> what is the minimum resource allocation for th= e cluster?
3) yarn-resource-manager-log =A0(while startin= g resource manager "export YARN_ROOT_LOGGER=3DDEBUG,RFA").. I am = looking for debug logs..
4) On RM UI how much total cluster memory is reported (how many total= nodes). ( RM UI click on Cluster)
5) which scheduler you= are using? Capacity/Fair/FIFO
6) have you configured any user limits/ queue capacity? (please add d= etails).
7) All requests you are making at same priority = or with different priorities? (Ideally it will not matter but want to know)= .

Please let us know all the above details. Than= ks.


Thanks,
Omkar Joshi


On Fri, Sep 20, 2013 at 6:55 AM, Antoine= Vandecreme <antoine.vandecreme@nist.gov> wrote:
Hello Omkar,

Thanks for your reply.

Yes, all 4 points are corrects.
However, my application is requesting let say 100 containers on my cluster<= br> which has 40 containers slots.
So I expected to see all containers slots used but that is not the case.
Just in case it matters, it is the only application running on the server.<= br>
Thanks,
Antoine Vandecreme

On Thursday, September 19, 2013 04:49:36 PM Omkar Joshi wrote:
> Hi,
>
> Let me clarify few things.
> 1) you are making container requests which are not explicitly looking = for
> certain nodes. (No white listing).
> 2) All nodes are identical in terms of resources (memory/cores) and ev= ery
> container requires same amount of resources.
> 3) All nodes have capacity to run say 2 containers.
> 4) You have 20 nodes.
>
> Now if an application is running and is requesting 20 containers then = you
> can not say that you will get all on different nodes (uniformly
> distributed). It more depends on which node heartbeated to the Resourc= e
> manager at what time and how much memory is available with it and also= how
> many applications are present in queue and how much they are requestin= g at
> what request priorities. If it has say sufficient memory to run 2
> containers then they will get allocated (This allocation is quite comp= lex
> ..I am assuming very simple "*" reuqest). So you may see few= running 2, few
> running 1 where as few with 0 containers.
>
> I hope it clarifies your doubt.
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Sep 19, 2013 at 7:19 AM, Vandecreme, Antoine <
>
> antoine.vandecreme@nist= .gov> wrote:
> > =A0Hi all,
> >
> > I am working with Hadoop 2.0.5 (I plan to migrate to 2.1.0 soon).=
> > When I am starting a Job, I notice that some nodes are not used o= r
> > partially used.
> >
> > For example, if my nodes can hold 2 containers, I notice that som= e nodes
> > are not running any or just 1 while others are running 2.
> > All my nodes are configured the same way.
> >
> > Is this an expected behavior (maybe in case others jobs are start= ed) ?
> > Is there a configuration to change this behavior?
> >
> > Thanks,
> > Antoine


CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --047d7b3441d0d3b26b04e6d59071--