avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Hammerbacher <ham...@cloudera.com>
Subject Re: Decoding a single field in Avro data
Date Wed, 25 Nov 2009 10:37:00 GMT
Hey Steinar,

When reading data from an Avro schema, you can specify a subset of the
schema as the "expected" schema and only those fields will be unpacked. I
don't know the exact details of how this works, but I have requested
documentation: https://issues.apache.org/jira/browse/AVRO-235. Perhaps you
could vote up that issue?

Thanks,
Jeff

On Wed, Nov 25, 2009 at 2:19 AM, Steinar Knutsen <steinar@yahoo-inc.com>wrote:

> Hi,
>
> I work in vertical search in Yahoo!, we would like to use for an
> application
> where it would be extremely beneficial to be able to extract single fields
> from binary Avro data without unpacking the entire datagram. This is hard
> to
> do efficiently, as far as I can tell. Avro supports skip information, but
> it
> is somewhat inefficient to skip across a block of an array, a record or a
> map, if any of these contain a variable length object. The headers only
> contain the number of objects contained, not the length in bytes.
>
> Is the correct way to solve this kind of scenario to create a schema with
> extra meta data (which feels like kind of a kludge), or is there some other
> way? Hopefully, I have overlooked something.
>
> Regards,
> --
> Steinar Knutsen <steinar@yahoo-inc.com>
> Yahoo! Technologies Norway AS
>

Mime
View raw message