avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tatu Saloranta <tsalora...@gmail.com>
Subject Re: Avro + Snappy changing blocksize of snappy compression
Date Wed, 18 Apr 2012 21:27:33 GMT
On Wed, Apr 18, 2012 at 2:18 PM, Scott Carey <scottcarey@apache.org> wrote:
> Try a range from smaller block sizes (4k) and up.  256K is a larger block
> size than many compression codecs are sensitive to.

Agreed: most codecs only go up to 32k or 64k (in fact, Snappy may use
just 32k, not 64k).
Deflate doesn't benefit from above 64k either, nor does lzf.
The only codecs that I think use larger buffers are bzip and lzma;
both of which are typically way too slow to be used for streaming data
processing anyway.

So testing up to 64k is usually enough.

-+ Tatu +-

View raw message