hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5196) CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing
Date Tue, 03 Jun 2014 10:04:02 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016357#comment-14016357
] 

Wangda Tan commented on MAPREDUCE-5196:
---------------------------------------


Hi [~curino], 
While trying to understand this part of change, I've several questions, hope you could share
some idea of it,

my understand of the workflow, please forgive my current ignorance of this code,
1) CheckpointAMPreemptionPolicy will keep track of which containers (task-attempt) need to
be preempted
2) TaskAttemptListener will set AMFeedback.preempted to true, when AMPreemptionPolicy.isPreempted(TaskID)
returns true
3) Task get AMFeedback, and set mustPreempt. Task takes some action to do some checkpoints,
etc. And call umbilical.preempted(taskId, taskStatus)
But a question here, I found the Task do nothing except printing some logs after mustPreempt
set, is it expected behavior? 
If it's expected behavior, the task will keep running until it completed or killed by NM,
resource cannot be proactively released by this task. I think we should call umbilical.preempted
when we found mustPreempt is true, correct?

Another question is, I found in Task.java
{code}
  public void done(TaskUmbilicalProtocol umbilical,
                   TaskReporter reporter
                   ) throws IOException, InterruptedException {
    updateCounters();
    if (taskStatus.getRunState() == TaskStatus.State.PREEMPTED ) {
      // If we are preempted, do no output promotion; signal done and exit
      committer.commitTask(taskContext);
      umbilical.preempted(taskId, taskStatus);
      taskDone.set(true);
      reporter.stopCommunicationThread();
      return;
    }
    ...
  }
{code}
It relies on taskStatus.getRunState() == PREEMPTED, but I found nobody set taskStatus.runState
to PREEMPTED. Could you please tell me which part of code set taskStatus. If nobody sets runState
to PREEMPTED, it cannot invoke umbilical.preempted properly.

Thanks,
Wangda


> CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing 
> ------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5196
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5196
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am, mrv2
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>             Fix For: 3.0.0
>
>         Attachments: MAPREDUCE-5196.1.patch, MAPREDUCE-5196.2.patch, MAPREDUCE-5196.3.patch,
MAPREDUCE-5196.patch, MAPREDUCE-5196.patch
>
>
> This JIRA tracks a checkpoint-based AM preemption policy. The policy handles propagation
of the preemption requests received from the RM to the appropriate tasks, and bookeeping of
checkpoints. Actual checkpointing of the task state is handled in upcoming JIRAs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message