hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4962) support filling up containers on node one by one
Date Fri, 15 Apr 2016 15:52:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15243136#comment-15243136
] 

Daniel Templeton commented on YARN-4962:
----------------------------------------

This is a common issue in mixed node clusters.  The typical solution is to load the cluster
"from different ends."  If the cluster has nodes of type A and type B (say, regular and big
memory) and jobs of type a and b (where type b jobs need type B machines), the scheduler should
schedule type a jobs to type A nodes, spilling into type B nodes only when there are no more
type A nodes available.  Type b jobs only run on type B nodes.

In Grid Engine, the way that's implemented is with node labels.  All type B nodes are labeled
as such.  All type b jobs are submitted with a hard request for type B nodes (hard = requires).
 All other jobs are submitted with a soft request for !type B nodes (soft = preference).

>  support filling up containers on node one by one
> -------------------------------------------------
>
>                 Key: YARN-4962
>                 URL: https://issues.apache.org/jira/browse/YARN-4962
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: sandflee
>
> we had a gpu cluster, jobs with bigger resource request couldn't be satisfied for node
is running the jobs with smaller resource request.  we didn't open reserve system because
gpu jobs may run days or weeks. we expect scheduler allocate containers to fill the node ,
then there will be resource to run   jobs with big resource request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message