crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-311) Expose injection of FileNamingScheme into AvroPathPerKeyTarget
Date Thu, 12 Dec 2013 04:46:09 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846062#comment-13846062
] 

Micah Whitacre commented on CRUNCH-311:
---------------------------------------

Looking at the code..

{code}
    Path basePath = new Path(getOutputPath(taskAttemptContext), conf.get("mapreduce.output.basename",
"part"));
    return new AvroFilePerKeyRecordWriter<T>(basePath, getUniqueFile(taskAttemptContext,
"part", ".avro"), conf);
{code}

I essentially want to override the "part" in the second line with an injectable value.  In
normal Hadoop configuration[1] this would be done through conf by setting the "mapreduce.output.basename"
but it looks like we are overriding that value for the basePath.  We could add a new config
value for that as a possibility.

[1] - https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java#L287

> Expose injection of FileNamingScheme into AvroPathPerKeyTarget
> --------------------------------------------------------------
>
>                 Key: CRUNCH-311
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-311
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.9.0
>            Reporter: Micah Whitacre
>            Assignee: Micah Whitacre
>             Fix For: 0.9.0
>
>         Attachments: CRUNCH-311.patch
>
>
> Similar to AvroFileTarget, injection of custom FileNamingScheme's should be supported
on the brand new AvroPathPerKeyTarget.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message