pig-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <sc...@richrelevance.com>
Subject Re: comments appreciated for pig AvroStorage
Date Wed, 01 Dec 2010 18:16:37 GMT
There are two other JIRAs with alternate Avro<-->Pig implementations with different feature

https://issues.apache.org/jira/browse/PIG-794 aims to use Avro internally within Pig for efficiency,
including intermediate serializatoin.

https://issues.apache.org/jira/browse/AVRO-592 has the same goals that your patch does, but
has fewer restrictions on what can and can't be written/read.  It supports writing any Pig
schema and reading it back in, but only reading a subset of Avro schemas (non-recursive; I
may add unions later).  With a little more work it could support intermediate serialization
for pig as well.   Longer term goals include being able to use AvroStorage along with a Hive
AvroSerDe on the same data, supporting projection, and supporting partitioning.

I've been hoping to finish up AVRO-592 but am currently busy with other things.


On Nov 30, 2010, at 9:05 PM, Lin Guo wrote:

> Hi,
> We'd like to patch our pig AvroStorage function and
> would highly appreciate any kinds of comments.
> doc:
> http://snaprojects.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data
> jira:
> https://issues.apache.org/jira/browse/PIG-1748
> Many thanks,
> Lin

View raw message