avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1243) Avro support for all compression codecs
Date Fri, 08 Feb 2013 17:07:16 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574627#comment-13574627
] 

Doug Cutting commented on AVRO-1243:
------------------------------------

One caution with this approach is that, as it stands, it will generate Avro data files that
can only be read by the Java implementation.  If we really want to support a particular codec
across languages we should add it to the Avro specification and implement it in C, C++, etc.
 LZO is close-enough in behavior to Snappy that I'm not sure it's worth adding to the spec,
but Bzip2 might be worth adding.
                
> Avro support for all compression codecs
> ---------------------------------------
>
>                 Key: AVRO-1243
>                 URL: https://issues.apache.org/jira/browse/AVRO-1243
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.7.3
>            Reporter: Ted Malaska
>            Priority: Minor
>         Attachments: AVRO-1243.not-ready.1.patch, AVRO-1243.not-ready.patch
>
>
> I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports
null, deflate, and snappy compression codecs.
> I would like to change the fromString method to use Class.forName(codec).newInstance();
after the codec was not found in the REGISTERED map but before the AvroRuntimeException is
thrown. 
> Here are some of my supporting thoughts
> 1. This should not interduce much slowness because it will only be called initialize.
> 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances
to the maven pom file.
> 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat
to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message