hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5196) CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing
Date Thu, 09 Jan 2014 17:24:52 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13866815#comment-13866815

Jason Lowe commented on MAPREDUCE-5196:

I recently stumbled across the fact that this removed the ping() method from the umbilical
and folded it into statusUpdate.  That's probably OK, but I noticed that it also changed the
semantic behavior of what a ping means.  Previously pings would not be considered making progress,
but now they are.  That's a problem if a task gets "stuck" where it's still connected to the
AM but not really making any progress.  The status updater thread will be happy to continue
pinging the AM yet the task has no hope of succeeding.  The progress watchdog in the AM ends
up being defeated and is only useful when the task can't contact the AM at all (e.g.: network
cut, overall process suspended, or crashed).

Was treating the ping as progress intentional and I'm missing something, or is this a bug?

> CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing 
> ------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5196
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5196
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am, mrv2
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>             Fix For: 3.0.0
>         Attachments: MAPREDUCE-5196.1.patch, MAPREDUCE-5196.2.patch, MAPREDUCE-5196.3.patch,
MAPREDUCE-5196.patch, MAPREDUCE-5196.patch
> This JIRA tracks a checkpoint-based AM preemption policy. The policy handles propagation
of the preemption requests received from the RM to the appropriate tasks, and bookeeping of
checkpoints. Actual checkpointing of the task state is handled in upcoming JIRAs.

This message was sent by Atlassian JIRA

View raw message