hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5196) CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing
Date Wed, 04 Jun 2014 15:23:02 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017761#comment-14017761

Carlo Curino commented on MAPREDUCE-5196:

Answering Wangda first:

For preemption we had to chop a very large set of changes in patches which are being slowly
pushed in (and churn in trunk made some this problematic).

I think your summary is correct. This patch does only propagate the information from the AM
to the Tasks. The tasks in turn only log the information. 
The actual checkpointing and release of resources is part of MAPREDUCE-5269. That patch used
to work on trunk, but has now some issues, 
and Augusto Souza is looking to fix back into shape, if you have cycles to look at it, help
on this is welcome. 

TaskStatus.State.PREEMPTED is set in MAPREDUCE-5269. Again this oddities are by-product of
separating a very large chunk of changes in more 
digestible-sized patches.

The idea of this patch is to fix the wiring so that the tasks knows about preemption, but
not change the behavior quite yet.

> CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing 
> ------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5196
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5196
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am, mrv2
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>             Fix For: 3.0.0
>         Attachments: MAPREDUCE-5196.1.patch, MAPREDUCE-5196.2.patch, MAPREDUCE-5196.3.patch,
MAPREDUCE-5196.patch, MAPREDUCE-5196.patch
> This JIRA tracks a checkpoint-based AM preemption policy. The policy handles propagation
of the preemption requests received from the RM to the appropriate tasks, and bookeeping of
checkpoints. Actual checkpointing of the task state is handled in upcoming JIRAs.

This message was sent by Atlassian JIRA

View raw message