pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Russell Jurney (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3015) Rewrite of AvroStorage
Date Tue, 11 Dec 2012 23:37:21 GMT

    [ https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529485#comment-13529485
] 

Russell Jurney commented on PIG-3015:
-------------------------------------

Pardon the comment blitz:

{noformat}
Russells-MacBook-Pro:pig-trunk rjurney$ grep -iR 'org.apache.avro.io.DatumReader' *
PIG-3015.patch:+import org.apache.avro.io.DatumReader;
PIG-3015.patch:+import org.apache.avro.io.DatumReader;
Binary file build/classes/org/apache/pig/builtin/AvroStorage.class matches
Binary file build/classes/org/apache/pig/impl/util/avro/AvroArrayReader.class matches
Binary file build/classes/org/apache/pig/impl/util/avro/AvroRecordReader.class matches
Binary file build/ivy/lib/Pig/avro-1.7.2.jar matches
Binary file build/ivy/lib/Pig/avro-tools-1.7.2.jar matches
Binary file build/pig-0.12.0-SNAPSHOT-withdependencies.jar matches
Binary file contrib/piggybank/java/build/classes/org/apache/pig/piggybank/storage/avro/AvroStorage.class
matches
Binary file contrib/piggybank/java/build/classes/org/apache/pig/piggybank/storage/avro/PigAvroRecordReader.class
matches
Binary file pig.jar matches
src/org/apache/pig/impl/util/avro/AvroArrayReader.java:import org.apache.avro.io.DatumReader;
src/org/apache/pig/impl/util/avro/AvroRecordReader.java:import org.apache.avro.io.DatumReader;
{noformat}

I'm puzzled... checking my classpath.
                
> Rewrite of AvroStorage
> ----------------------
>
>                 Key: PIG-3015
>                 URL: https://issues.apache.org/jira/browse/PIG-3015
>             Project: Pig
>          Issue Type: Improvement
>          Components: piggybank
>            Reporter: Joseph Adler
>            Assignee: Joseph Adler
>         Attachments: PIG-3015.patch
>
>
> The current AvroStorage implementation has a lot of issues: it requires old versions
of Avro, it copies data much more than needed, and it's verbose and complicated. (One pet
peeve of mine is that old versions of Avro don't support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the new implementation
is significantly faster, and the code is a lot simpler. Rewriting AvroStorage also enabled
me to implement support for Trevni (as TrevniStorage).
> I'm opening this ticket to facilitate discussion while I figure out the best way to contribute
the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message