hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-201) CapacityScheduler can take a very long time to schedule containers if requests are off cluster
Date Tue, 06 Nov 2012 16:58:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491613#comment-13491613
] 

Jason Lowe commented on YARN-201:
---------------------------------

I looked into why 1.x jobs don't have this issue.  I believe it's because of how JobInProgress
is resetting scheduling opportunities.  When it allocates a node-local or rack-local task
it resets the opportunities, but if it allocates a non-local task then it explicitly avoids
resetting the opportunities, with a comment stating as such.  YARN's CapacityScheduler resets
the scheduling opportunities for any container allocated.

So one potential fix is to emulate the 1.x behavior and not reset scheduling opportunities
when we end up assigning a non-local container.  Another is to have the CapacityScheduler
track the active nodes/racks and filter requests for nodes/racks that aren't in that list
when we normalize the ask list from the AM.  This would keep us from waiting around for scheduling
opportunities that will never help us meet locality for an ask that's off-cluster.
                
> CapacityScheduler can take a very long time to schedule containers if requests are off
cluster
> ----------------------------------------------------------------------------------------------
>
>                 Key: YARN-201
>                 URL: https://issues.apache.org/jira/browse/YARN-201
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 0.23.3, 2.0.1-alpha
>            Reporter: Jason Lowe
>
> When a user runs a job where one of the input files is a large file on another cluster,
the job can create many splits on nodes which are unreachable for computation from the current
cluster.  The off-switch delay logic in LeafQueue can cause the ResourceManager to allocate
containers for the job very slowly.  In one case the job was only getting one container every
23 seconds, and the queue had plenty of spare capacity.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message