Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 16 Feb 2017 15:28:42 +0000 (UTC)
From: "Jason Lowe (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13043082.1487108190000.100136.1487258922612@Atlassian.JIRA>
In-Reply-To: <JIRA.13043082.1487108190000@Atlassian.JIRA>
References: <JIRA.13043082.1487108190000@Atlassian.JIRA> <JIRA.13043082.1487108190918@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-6191) CapacityScheduler preemption by
 container priority can be problematic for MapReduce
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 16 Feb 2017 15:28:49 -0000


    [ https://issues.apache.org/jira/browse/YARN-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870131#comment-15870131 ] 

Jason Lowe commented on YARN-6191:
----------------------------------

Thanks, Chris!  Having the AM react to the preemption message in the heartbeat will definitely help a lot for common cases, even if it doesn't do any work-conserving logic and just kills the reducers.

However there's still an issue because the preemption message is too general.  For example, if the message says "going to preempt 60GB of resources" and the AM kills 10 reducers that are 6GB each on 6 different nodes, the RM can still kill the maps because the RM needed 60GB of contiguous resources.  Fixing that requires the preemption message to be more expressive/specific so the AM knows that its actions will indeed prevent the preemption of other containers.

I still wonder about the logic of preferring lower container priorities regardless of how long they've been running.  I'm not sure container priority always translates well to how important a container is to the application, and we might be better served by preferring to minimize total lost work regardless of container priority.

> CapacityScheduler preemption by container priority can be problematic for MapReduce
> -----------------------------------------------------------------------------------
>
>                 Key: YARN-6191
>                 URL: https://issues.apache.org/jira/browse/YARN-6191
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>            Reporter: Jason Lowe
>
> A MapReduce job with thousands of reducers and just a couple of maps left to go was running in a preemptable queue.  Periodically other queues would get busy and the RM would preempt some resources from the job, but it _always_ picked the job's map tasks first because they use the lowest priority containers.  Even though the reducers had a shorter running time, most were spared but the maps were always shot.  Since the map tasks ran for a longer time than the preemption period, the job was in a perpetual preemption loop.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org