hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antoine Vandecreme <antoine.vandecr...@nist.gov>
Subject Re: How to make hadoop use all nodes?
Date Mon, 23 Sep 2013 18:14:34 GMT
Hi Omkar,

>(which has 40 containers slots.) >> for total cluster?
Yes, it was just an hypotetical value though.
Below are my real configurations.

>1) yarn-site.xml -> what is the resource memory configured for per node?
12288mb

>2) yarn-site.xml -> what is the minimum resource allocation for the 
cluster?
1024mb min
12288mb max

I also have those memory configurations in mapred-site.xml :
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>5000</value>
  </property>

  <property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx4g -Djava.awt.headless=true</value>
  </property>

  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>5000</value>
  </property>

  <property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx4g -Djava.awt.headless=true</value>
  </property>

>3) yarn-resource-manager-log  (while starting resource manager "export 
YARN_ROOT_LOGGER=DEBUG,RFA").. I am looking for debug logs..
The resulting log is really verbose. Are you searching for something in 
particular?

>4) On RM UI how much total cluster memory is reported (how many total 
nodes). ( RM UI click on Cluster)
So I have 58 active nodes and total memory reported is 696GB which is 
58x12 as expected.
I have 93 containers running instead of 116 I would expect (my job has 
2046 maps so it could use all 116 containers).

Here is a copy past of what I have in the scheduler tab:


*Queue State: *
RUNNING 
*Used Capacity: *
99.4% 
*Absolute Capacity: *
100.0% 
*Absolute Max Capacity: *
100.0% 
*Used Resources: *

*Num Active Applications: *
1 
*Num Pending Applications: *
0 
*Num Containers: *
139 
*Max Applications: *
10000 
*Max Applications Per User: *
10000 
*Max Active Applications: *
70 
*Max Active Applications Per User: *
70 
*Configured Capacity: *
100.0% 
*Configured Max Capacity: *
100.0% 
*Configured Minimum User Limit Percent: *
100% 
*Configured User Limit Factor: *
1.0 
*Active users: *
xxx <Memory: 708608 (100.00%), vCores: 139 (100.00%), Active Apps: 1, 
Pending Apps: 0>


I don't know where the 139 containers value is comming from.

>5) which scheduler you are using? Capacity/Fair/FIFO
I did not set yarn.resourcemanager.scheduler.class so apparently the 
default is Capacity.

>6) have you configured any user limits/ queue capacity? (please add 
details).
No.

>7) All requests you are making at same priority or with different priorities? 
(Ideally it will not matter but want to know).
I don't set any priority.

Thanks for your help.

Antoine Vandecreme

On Friday, September 20, 2013 12:20:38 PM Omkar Joshi wrote:
> Hi,
> 
> few more questions
> 
> (which has 40 containers slots.) >> for total cluster? Please give below
> details
> 
> for cluster
> 1) yarn-site.xml -> what is the resource memory configured for per node?
> 2) yarn-site.xml -> what is the minimum resource allocation for the 
cluster?
> 3) yarn-resource-manager-log  (while starting resource manager "export
> YARN_ROOT_LOGGER=DEBUG,RFA").. I am looking for debug logs..
> 4) On RM UI how much total cluster memory is reported (how many total
> nodes). ( RM UI click on Cluster)
> 5) which scheduler you are using? Capacity/Fair/FIFO
> 6) have you configured any user limits/ queue capacity? (please add
> details).
> 7) All requests you are making at same priority or with different
> priorities? (Ideally it will not matter but want to know).
> 
> Please let us know all the above details. Thanks.
> 
> 
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
> 
> 
> On Fri, Sep 20, 2013 at 6:55 AM, Antoine Vandecreme <
> 
> antoine.vandecreme@nist.gov> wrote:
> > Hello Omkar,
> > 
> > Thanks for your reply.
> > 
> > Yes, all 4 points are corrects.
Mime
View raw message