avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Hauser <ewhau...@gmail.com>
Subject Compression and splittable Avro files in Hadoop
Date Sat, 01 Oct 2011 01:42:32 GMT
A coworker and I were having a conversation today about choosing a
compression algorithm for some data we are storing in Hadoop.  We have
been using (https://github.com/tomslabs/avro-utils) for our Map/Reduce
jobs and Haivvreo for integration with Hive.  By default, the
avro-utils OutputFormat uses deflate compression.  Even though
default/zlib/gzip files are not splittable, we decided that Avro data
files are always splittable because individual blocks within the file
are compressed instead of the entire file.

Is this accurate?  Thanks.

Mime
View raw message