hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@huawei.com>
Subject RE: Node manager contributing to one queue's resources
Date Thu, 26 Feb 2015 09:13:25 GMT
Hi

If you are using CapacityScheduler, can you try using DominantResourceCalculator i.e configuring
below property value in capacity-scheduler.xml file.

  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource. DominantResourceCalculator </value>
  </property>

The basic Idea it works is as follows  ‘if user A runs CPU-heavy tasks and user B runs memory-heavy
tasks, it attempts to equalize CPU share of user A with Memory-share of user B’

See Java Doc
https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java

I think this may help you!!!

Thanks & Regards
Rohith Sharma K S

From: twinkle sachdeva [mailto:twinkle.sachdeva@gmail.com]
Sent: 26 February 2015 14:05
To: USers Hadoop
Subject: Node manager contributing to one queue's resources

Hi,

I have to run two kind of applications, one requiring less cores but more memory ( Application_High_Mem)
and another application which requires more cores but less memory ( Application_High_Core).

I can use specific queues to submit them to, but that can lead to one node contributing to
only one one such application and having some part of resources as idle.

Is there a way, let's say extending concept of queues at node manager level to do this or
some other way, in which i can achieve it in YARN?

Thanks,
Twinkle
Mime
View raw message