Mailing-List: contact user-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@avro.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CACBYqwRXf_KkM5DAJ0dwWVrF+d-pKjDBpvesroxwO3xw0pXVRA@mail.gmail.com>
References: 
 <CACBYqwRXf_KkM5DAJ0dwWVrF+d-pKjDBpvesroxwO3xw0pXVRA@mail.gmail.com>
Date: Thu, 5 Jul 2012 10:24:00 -0700
Message-ID: 
 <CALEq1Z-_czLyGtzMG+NnsjNKcc3=5BQwpDieZjH1uKnF5XthKg@mail.gmail.com>
Subject: Re: Avro file size is too big
From: Doug Cutting <cutting@apache.org>
To: user@avro.apache.org
Content-Type: text/plain; charset=UTF-8

Rusian,

This is unexpected.  Perhaps we can understand it if we have more information.

What Writable class are you using for keys and values in the SequenceFile?

What schema are you using in the Avro data file?

Can you provide small sample files of each and/or code that will reproduce this?

Thanks,

Doug

On Wed, Jul 4, 2012 at 6:32 AM, Ruslan Al-Fakikh <metaruslan@gmail.com> wrote:
> Hello,
>
> In my organization currently we are evaluating Avro as a format. Our
> concern is file size. I've done some comparisons of a piece of our
> data.
> Say we have sequence files, compressed. The payload (values) are just
> lines. As far as I know we use line number as keys and we use the
> default codec for compression inside sequence files. The size is 1.6G,
> when I put it to avro with deflate codec with deflate level 9 it
> becomes 2.2G.
> This is interesting, because the values in seq files are just string,
> but Avro has a normal schema with primitive types. And those are kept
> binary. Shouldn't Avro be less in size?
> Also I took another dataset which is 28G (gzip files, plain
> tab-delimited text, don't know what is the deflate level) and put it
> to Avro and it became 38G
> Why Avro is so big in size? Am I missing some size optimization?
>
> Thanks in advance!