avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1243) Avro support for all compression codecs
Date Fri, 08 Feb 2013 23:47:15 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574953#comment-13574953
] 

Doug Cutting commented on AVRO-1243:
------------------------------------

gzip files use the deflate codec, so we're already handling that.

The latest patch looks like a good approach.

Comments:
 - There's a cut/paste error in the bzip2Codec() javadoc.
 - There are a lot of whitespace-only changes in the patch.
 - The fromHadoopString() method should probably live in the avro-mapred module.
                
> Avro support for all compression codecs
> ---------------------------------------
>
>                 Key: AVRO-1243
>                 URL: https://issues.apache.org/jira/browse/AVRO-1243
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.7.3
>            Reporter: Ted Malaska
>            Priority: Minor
>         Attachments: AVRO-1243.not-ready.1.patch, AVRO-1243.not-ready.2.patch, AVRO-1243.not-ready.patch
>
>
> I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports
null, deflate, and snappy compression codecs.
> I would like to change the fromString method to use Class.forName(codec).newInstance();
after the codec was not found in the REGISTERED map but before the AvroRuntimeException is
thrown. 
> Here are some of my supporting thoughts
> 1. This should not interduce much slowness because it will only be called initialize.
> 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances
to the maven pom file.
> 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat
to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message