hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3258) FileOutputFormat should have a method to create custom files under the outputdir with a unique name per task to avoid name collision
Date Thu, 24 Apr 2008 04:13:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591913#action_12591913
] 

Alejandro Abdelnur commented on HADOOP-3258:
--------------------------------------------

Owen,

It would not be death code, the intention is to refactor Hadoop-3149 to use this patch.

I've submitted this patch because I've found our pr oject requiring this functionality over
and over and I've thought it would be useful for other folks as well.

If nobody sees value and you -1 again I'll close this issue as won't fix.


> FileOutputFormat should have a method to create custom files under the outputdir with
a unique name per task to avoid name collision
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3258
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3258
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>         Environment: all
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.18.0
>
>         Attachments: patch-3258.txt, patch-3258.txt
>
>
> Currently, if a M/R code creates a file, it is the responsibility of the M/R code to
avoid file name collisions from different tasks.
> Hadoop should provide an API that creates unique file names based on the task type (map
or reduce) and the task ID. Similarly to how output files, part-#####, are created.
> The proposed patch adds 2 static methods to the {{FileOutputFormat}}
> {nofomat}
>   public static String getUniqueName(JobConf conf, String name);
>   public static Path getPathForCustomFile(JobConf conf, String name);
> {nofomat}
> The first one adds task type and task ID to the given name.
> The second gives a PATH to a file in the working outputdir using a file name namespaced
by the first method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message