hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-624) Support gang scheduling in the AM RM protocol
Date Mon, 12 Aug 2013 19:06:54 GMT

    [ https://issues.apache.org/jira/browse/YARN-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13737246#comment-13737246
] 

Robert Joseph Evans commented on YARN-624:
------------------------------------------

Carlo,

I do have to be total honest here too.  Gang scheduling is an interesting feature, but it
is not the highest on my priority list.  In the short to medium term I am fine with launching
containers and letting them wait until there are enough to run the entire topology.  I am
also fine if I have to configure my queues appropriately to avoid deadlock/preemption in the
short to medium term. If it is too difficult to have preemption be gang aware we can skip
it, and ask users to configure the queue appropriately. I just want to be sure that whatever
route we start down does not lock us into a model that will prevent me from getting to my
end goal. 
                
> Support gang scheduling in the AM RM protocol
> ---------------------------------------------
>
>                 Key: YARN-624
>                 URL: https://issues.apache.org/jira/browse/YARN-624
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, scheduler
>    Affects Versions: 2.0.4-alpha
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>
> Per discussion on YARN-392 and elsewhere, gang scheduling, in which a scheduler runs
a set of tasks when they can all be run at the same time, would be a useful feature for YARN
schedulers to support.
> Currently, AMs can approximate this by holding on to containers until they get all the
ones they need.  However, this lends itself to deadlocks when different AMs are waiting on
the same containers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message