hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4462) Scheduler should prevent certain application from being preempted
Date Wed, 16 Dec 2015 03:39:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059444#comment-15059444

Sunil G commented on YARN-4462:

Hi [~Tao Jie]
Trying to understand the problem here. SO you mentioned that few applications were preempted
and became slow, correct?. This can happen only if the resource usage went across the desired
configured limit for that Queue. And there were demands (some applications waiting to get
containers) to an under-serving queue. This would have resulted in preemption.

In preemption, we will be considering the last submitted applications as first candidate to
preempt (within each application, will consider the lowest priority containers) within queue.
Few tuning could be done to avoid some sudden trigger of preemption such as "dead-zone" etc.
Pls check the preemption related configurations for same.

Now coming to the suggestion, I fee its kind of a hack over preemption. Preemption policy
now chooses the apps/containers based on above explained criteria. And if we have more apps
labelled with "do-not-preempt", I feel it may not give a fair preemption across applications
in a queue. I am not very sure whether we need to give some extra power to a queue/user to
save some v.imp apps. But needs to see how far it will help in reality. 

> Scheduler should prevent certain application from being preempted
> -----------------------------------------------------------------
>                 Key: YARN-4462
>                 URL: https://issues.apache.org/jira/browse/YARN-4462
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 2.6.0
>            Reporter: Tao Jie
> When scheduler preemption is enabled, applications could be preempted if they obtain
resource over they should take. 
> When a mapreduce application is preempted some resource, it just runs slower. However,
when the preempted application is a long-run service, such as tomcat running in slider, the
service would fail.
> So we should have a flag for application to indicate the scheduler that those application
should not be preempted. 

This message was sent by Atlassian JIRA

View raw message