hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3150) Move task file promotion into the task
Date Wed, 04 Jun 2008 19:50:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602435#action_12602435

Owen O'Malley commented on HADOOP-3150:

Which leads back to the fact that I'm pretty uncomfortable with such a big change to the APIs
this close the freeze date. The current OutputFormat method names are very file system specific.
I'd prefer something like:

public class JobContext {
  public JobConf getJobConf();

public class TaskAttemptContext {
  public TaskAttemptID getTaskAttemptID();
  public JobConf getJobConf();
  public Progressable getProgressable();

public abstract class OutputFormat {
  abstract public RecordWriter<K, V> getRecordWriter(TaskAttemptContext context
             ) throws IOException;
  abstract public void checkOutputSpecs(JobContext context) throws IOException;

  public abstract setupJob(JobContext context) throws IOException;
  public abstract cleanupJob(JobContext context, boolean promote) throws IOException;
  public abstract setupTask(TaskAttemptContext context) throws IOException;
  public abstract cleanupTask(TaskAttemptContext context, boolean promote) throws IOException;

At that point, you might as well move OutputFormat over to org.apache.hadoop.mapreduce and
deprecate the old interface.

I don't think having the TaskTracker rpc handler (handling a call from the task) block on
a call to the JobTracker is workable.

> Move task file promotion into the task
> --------------------------------------
>                 Key: HADOOP-3150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3150
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.18.0
>         Attachments: 3150.patch
> We need to move the task file promotion from the JobTracker to the Task and move it down
into the output format.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message