Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Tue, 6 Nov 2012 16:58:13 +0000 (UTC)
From: "Jason Lowe (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <277373097.75107.1352221093713.JavaMail.jiratomcat@arcas>
In-Reply-To: <1360692052.69158.1352130492678.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (YARN-201) CapacityScheduler can take a very
 long time to schedule containers if requests are off cluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491613#comment-13491613 ] 

Jason Lowe commented on YARN-201:
---------------------------------

I looked into why 1.x jobs don't have this issue.  I believe it's because of how JobInProgress is resetting scheduling opportunities.  When it allocates a node-local or rack-local task it resets the opportunities, but if it allocates a non-local task then it explicitly avoids resetting the opportunities, with a comment stating as such.  YARN's CapacityScheduler resets the scheduling opportunities for any container allocated.

So one potential fix is to emulate the 1.x behavior and not reset scheduling opportunities when we end up assigning a non-local container.  Another is to have the CapacityScheduler track the active nodes/racks and filter requests for nodes/racks that aren't in that list when we normalize the ask list from the AM.  This would keep us from waiting around for scheduling opportunities that will never help us meet locality for an ask that's off-cluster.
                
> CapacityScheduler can take a very long time to schedule containers if requests are off cluster
> ----------------------------------------------------------------------------------------------
>
>                 Key: YARN-201
>                 URL: https://issues.apache.org/jira/browse/YARN-201
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 0.23.3, 2.0.1-alpha
>            Reporter: Jason Lowe
>
> When a user runs a job where one of the input files is a large file on another cluster, the job can create many splits on nodes which are unreachable for computation from the current cluster.  The off-switch delay logic in LeafQueue can cause the ResourceManager to allocate containers for the job very slowly.  In one case the job was only getting one container every 23 seconds, and the queue had plenty of spare capacity.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira