avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Object Container Files format questions
Date Mon, 04 May 2009 17:43:00 GMT
Matt Massie wrote:
> (1) Why is a meta block not required to immediately follow the header?  It
> seems counter-intuitive to me that the requirement is for a meta block at
> the end of the file.  I would expect that we would want the reader to know
> the meta data before any data blocks are sent (especially if you want to
> transfer/decode a file on the fly over the wire).

I wanted to support append. The idea is that the meta-block is written 
each time the file is flushed, and that the file is valid through the 
last meta-block.  If a program crashes without flushing, then the file 
might be truncated to its last meta-block, and then new data could be 

I did not have streaming in mind.  The meta-block was intended to be 
retroactive.  I contains the total number of entries.  If the file's 
schema is a union, one might add new entries into the union as the file 
is written, as new types are added, writing the full union when the file 
is flushed.

To better ssupport streaming we might require that the metadata be 
written every so many bytes, so that a streaming application could 
buffer until it sees a metadata entry, then process the buffered items. 
  Could that work?  Note that automatic periodic metadata writes would 
also facilitate crash recovery, since one would know how far from the 
end of the file to begin scanning.

> (2) Why are we using a 16-byte UUID block delimiter to mark block boundaries
> instead of using say Consistent Overhead Byte Stuffing (COBS)?

In part because I didn't yet know about COBS when I implemented this 
container.  (You're the second person to ask this.)  Perhaps we should 
instead use COBS.  The only potential disadvantage I see is that COBS 
seems to require byte-by-byte processing.  When projecting records to a 
subset schema, we've seen huge speedups if we skip through data by 
chunks, passing over strings, blobs, etc. just incrementing the file 
pointer.  So I worry that COBS, while quite fast, might add a 
significant cost to such skipping.  Compression also requires 
byte-by-byte processing, but provides more tangible value than COBS.  So 
COBS would need to add truly negligible CPU overhead, which it might.

> COBs
> also allows for in-place encoding and decoding with very little overhead and
> copying (see the paper for test results).
[ ... ]

Thanks for all your COBS analysis!  It does seem attractive.  Skimming 
the paper, I see mostly space overhead benchmarks, not CPU overhead.  In 
projection benchmarks, we saw a 4x speedup from just skipping strings. 
AVRO-25 proposes to make entire arrays and maps skippable, which should, 
e.g., speed the projection of a single field from large, complex records 
by 10x or more.  But if compression is always used, perhaps the right 
comparison is with something like LZO.  If COBS is negligible next to 
LZO, that could be good enough.

Hopefully we'll develop a benchmark suite for Avro that includes 
projection & compression, so we can more easily evaluate things like this.


View raw message