avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Malaska (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1243) Avro support for all compression codecs
Date Fri, 08 Feb 2013 19:01:13 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574729#comment-13574729
] 

Ted Malaska commented on AVRO-1243:
-----------------------------------

BTW just wondering.  Should GZip in or out?  I understand why LZO should be out, but I was
just wondering what should be the fate of GZip.

I know deflate is faster then GZip but I've read it doesn't compress as well.  I also know
the GZip reads very fast, very comparable to Snappy read speeds.  So for write once and read
many applications GZip is a nice pick.


                
> Avro support for all compression codecs
> ---------------------------------------
>
>                 Key: AVRO-1243
>                 URL: https://issues.apache.org/jira/browse/AVRO-1243
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.7.3
>            Reporter: Ted Malaska
>            Priority: Minor
>         Attachments: AVRO-1243.not-ready.1.patch, AVRO-1243.not-ready.patch
>
>
> I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports
null, deflate, and snappy compression codecs.
> I would like to change the fromString method to use Class.forName(codec).newInstance();
after the codec was not found in the REGISTERED map but before the AvroRuntimeException is
thrown. 
> Here are some of my supporting thoughts
> 1. This should not interduce much slowness because it will only be called initialize.
> 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances
to the maven pom file.
> 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat
to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message