hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antoine Vandecreme <antoine.vandecr...@nist.gov>
Subject Re: How to make hadoop use all nodes?
Date Fri, 20 Sep 2013 13:55:17 GMT
Hello Omkar,

Thanks for your reply.

Yes, all 4 points are corrects.
However, my application is requesting let say 100 containers on my cluster 
which has 40 containers slots.
So I expected to see all containers slots used but that is not the case.

Just in case it matters, it is the only application running on the server.

Thanks,
Antoine Vandecreme

On Thursday, September 19, 2013 04:49:36 PM Omkar Joshi wrote:
> Hi,
> 
> Let me clarify few things.
> 1) you are making container requests which are not explicitly looking for
> certain nodes. (No white listing).
> 2) All nodes are identical in terms of resources (memory/cores) and every
> container requires same amount of resources.
> 3) All nodes have capacity to run say 2 containers.
> 4) You have 20 nodes.
> 
> Now if an application is running and is requesting 20 containers then you
> can not say that you will get all on different nodes (uniformly
> distributed). It more depends on which node heartbeated to the Resource
> manager at what time and how much memory is available with it and also how
> many applications are present in queue and how much they are requesting at
> what request priorities. If it has say sufficient memory to run 2
> containers then they will get allocated (This allocation is quite complex
> ..I am assuming very simple "*" reuqest). So you may see few running 2, few
> running 1 where as few with 0 containers.
> 
> I hope it clarifies your doubt.
> 
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
> 
> 
> On Thu, Sep 19, 2013 at 7:19 AM, Vandecreme, Antoine <
> 
> antoine.vandecreme@nist.gov> wrote:
> >  Hi all,
> > 
> > I am working with Hadoop 2.0.5 (I plan to migrate to 2.1.0 soon).
> > When I am starting a Job, I notice that some nodes are not used or
> > partially used.
> > 
> > For example, if my nodes can hold 2 containers, I notice that some nodes
> > are not running any or just 1 while others are running 2.
> > All my nodes are configured the same way.
> > 
> > Is this an expected behavior (maybe in case others jobs are started) ?
> > Is there a configuration to change this behavior?
> > 
> > Thanks,
> > Antoine

Mime
View raw message