hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3150) Move task file promotion into the task
Date Mon, 21 Jul 2008 12:37:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615231#action_12615231

Amareshwari Sriramadasu commented on HADOOP-3150:

This sounds good.

But some clarifications:

1. Shouldn't we move the apis
  public static Path getWorkOutputPath(JobConf conf);
  static void setWorkOutputPath(JobConf conf, Path outputDir);
from FileOutputFormat to FileOutputCommitter and deprecate  _getWorkOutputPath(JobConf conf)_
in FileOutputFormat ?

2. Since we do not do commit/abort of output at Job level and we do cleanup of temporary stuff,
insteadof apis {commit/abort}Job, we should have cleanupJob.  

bq. The FileOutputFormat would check if the OutputCommitter is a FileOutputCommitter and if
so, it would use the getWorkPath from it.
Owen, do you say that fileOutputCommitter.getWorkPath(basePath) should be called from fileOutputFormat.getRecordWriter()

> Move task file promotion into the task
> --------------------------------------
>                 Key: HADOOP-3150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3150
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Owen O'Malley
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.19.0
>         Attachments: 3150.patch, patch-3150.txt, patch-3150.txt
> We need to move the task file promotion from the JobTracker to the Task and move it down
into the output format.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message