pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheolsoo Park (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3015) Rewrite of AvroStorage
Date Tue, 06 Nov 2012 06:06:12 GMT

    [ https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491231#comment-13491231

Cheolsoo Park commented on PIG-3015:

Hi Joseph,

To answer your questions:

1) If I am not mistaken, o.a.p.impl.builtin is for internal built-in UDFs. I don't know exactly
what your helper classes are like, but would o.a.p.impl.util be a better place?

Looking at the package tree, I also noticed that there is an *.impl.util package for each
sub-component of Pig. So if your helper classes are AvroStorage-specific, you may want to
create two new packages called o.a.p.hadoop.avro and o.a.p.hadoop.avro.util, and add AvroStorage
to hadoop.avro and helper classes to hadoop.avro.util respectively.

Please anyone correct me if I am wrong here. I am a new committer. :-)

2) What you propose sounds good to me.

> Rewrite of AvroStorage
> ----------------------
>                 Key: PIG-3015
>                 URL: https://issues.apache.org/jira/browse/PIG-3015
>             Project: Pig
>          Issue Type: Improvement
>          Components: piggybank
>            Reporter: Joseph Adler
>            Assignee: Joseph Adler
> The current AvroStorage implementation has a lot of issues: it requires old versions
of Avro, it copies data much more than needed, and it's verbose and complicated. (One pet
peeve of mine is that old versions of Avro don't support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the new implementation
is significantly faster, and the code is a lot simpler. Rewriting AvroStorage also enabled
me to implement support for Trevni.
> I'm opening this ticket to facilitate discussion while I figure out the best way to contribute
the changes back to Apache.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message