hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-568) FairScheduler: support for work-preserving preemption
Date Wed, 08 May 2013 02:05:17 GMT

    [ https://issues.apache.org/jira/browse/YARN-568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651550#comment-13651550
] 

Carlo Curino commented on YARN-568:
-----------------------------------

Sandy, I ran, in a small 10 machines cluster, the FS patched with your latest edit. 

I confirmed the previous correct behavior we had in our patch, i.e., the scheduler logged
preemption and killing actions. 
I also ran it with our "preemptable MR" with checkpointing as an AM, and the preemption messages
were propagated to the AM 
correctly (and checkpointing triggered). 

(Also we independently tested that the PreemptionContract, side of YARN-45 was exercised properly
by the CapacityScheduler, so 
also YARN-567 which is now rolled into this seem in good shape). 

So I would say, this patch is good to go for me. 

As a separate consideration, I think we should work on good defaults parameters and good indications
on how to tune this preemption policy.

                
> FairScheduler: support for work-preserving preemption 
> ------------------------------------------------------
>
>                 Key: YARN-568
>                 URL: https://issues.apache.org/jira/browse/YARN-568
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-568-1.patch, YARN-568-2.patch, YARN-568-2.patch, YARN-568.patch,
YARN-568.patch
>
>
> In the attached patch, we modified  the FairScheduler to substitute its preemption-by-killling
with a work-preserving version of preemption (followed by killing if the AMs do not respond
quickly enough). This should allows to run preemption checking more often, but kill less often
(proper tuning to be investigated).  Depends on YARN-567 and YARN-45, is related to YARN-569.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message