hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From twinkle sachdeva <twinkle.sachd...@gmail.com>
Subject Re: Node manager contributing to one queue's resources
Date Fri, 27 Feb 2015 06:21:23 GMT
Thanks for the sharing another view point, will look around it.
But seems like, we might need something more specific.


On Thu, Feb 26, 2015 at 2:43 PM, Rohith Sharma K S <
rohithsharmaks@huawei.com> wrote:

>  Hi
>
>
>
> If you are using CapacityScheduler, can you try using
> DominantResourceCalculator i.e configuring below property value in
> capacity-scheduler.xml file.
>
>
>
>   <property>
>
>     <name>yarn.scheduler.capacity.resource-calculator</name>
>
>     <value>org.apache.hadoop.yarn.util.resource.
> DominantResourceCalculator </value>
>
>   </property>
>
>
>
> The basic Idea it works is as follows  ‘if user A runs CPU-heavy tasks
> and user B runs memory-heavy tasks, it attempts to equalize CPU share of
> user A with Memory-share of user B’
>
>
>
> See Java Doc
>
>
> https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java
>
>
>
> I think this may help you!!!
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
> *From:* twinkle sachdeva [mailto:twinkle.sachdeva@gmail.com]
> *Sent:* 26 February 2015 14:05
> *To:* USers Hadoop
> *Subject:* Node manager contributing to one queue's resources
>
>
>
> Hi,
>
>
>
> I have to run two kind of applications, one requiring less cores but more
> memory ( Application_High_Mem) and another application which requires more
> cores but less memory ( Application_High_Core).
>
>
>
> I can use specific queues to submit them to, but that can lead to one node
> contributing to only one one such application and having some part of
> resources as idle.
>
>
>
> Is there a way, let's say extending concept of queues at node manager
> level to do this or some other way, in which i can achieve it in YARN?
>
>
>
> Thanks,
>
> Twinkle
>

Mime
View raw message