pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Adler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3015) Rewrite of AvroStorage
Date Wed, 05 Dec 2012 17:51:58 GMT

    [ https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510629#comment-13510629

Joseph Adler commented on PIG-3015:

Hi Johannes,

As you probably know, the Avro specification limits the set of valid characters in names (see
http://avro.apache.org/docs/current/spec.html#Names). Names must

- start with [A-Za-z_]
- subsequently contain only [A-Za-z0-9_]

So double colons aren't allowed. PIG-2684 proposes using namespaces as the solution. I think
that's a poor choice; namespaces are often used for other purposes. Specifically, names spaces
are essential if you are writing complicated data processing software that processes multiple
types of avro serialized objects. In my experience, the avro schema and protocol compilers
produce much better, more usable code if you use name spaces.

There are two good workarounds:

- The Pig user can rename variables in a bag before storing the bag using AvroStorage
- The Pig user can manually specify the output schema before storing the bag with AvroStorage

So, here's a specific suggestion:

- By default, throw an exception if the pig schema contains a name with a double-colon and
the user does not specify an output schema
- Add an option to AvroStorage to transform double colons to something else. (Maybe double
underscores? Maybe storing them in the namespace?)

What do you think?

> Rewrite of AvroStorage
> ----------------------
>                 Key: PIG-3015
>                 URL: https://issues.apache.org/jira/browse/PIG-3015
>             Project: Pig
>          Issue Type: Improvement
>          Components: piggybank
>            Reporter: Joseph Adler
>            Assignee: Joseph Adler
>         Attachments: PIG-3015.patch
> The current AvroStorage implementation has a lot of issues: it requires old versions
of Avro, it copies data much more than needed, and it's verbose and complicated. (One pet
peeve of mine is that old versions of Avro don't support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the new implementation
is significantly faster, and the code is a lot simpler. Rewriting AvroStorage also enabled
me to implement support for Trevni.
> I'm opening this ticket to facilitate discussion while I figure out the best way to contribute
the changes back to Apache.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message