Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
User-Agent: Microsoft-MacOutlook/f.19.0.160817
Date: Wed, 25 Jan 2017 14:13:20 -0800
Subject: Re: Parquet tables with snappy compression
From: Gopal Vijayaraghavan <gopalv@apache.org>
Sender: Gopal Vijayaraghavan <gopal@hortonworks.com>
To: "user@hive.apache.org" <user@hive.apache.org>
Message-ID: <BFCCF5E4-CDA1-4DC5-8040-8136CB65935C@hortonworks.com>
Thread-Topic: Parquet tables with snappy compression
References: <CAJ3fcbAuNL5XiZionB3e=gMGO3GQpvMySpZGbrPk9HeO9Xkg=Q@mail.gmail.com>
In-Reply-To: <CAJ3fcbAuNL5XiZionB3e=gMGO3GQpvMySpZGbrPk9HeO9Xkg=Q@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="UTF-8"
Content-transfer-encoding: quoted-printable
archived-at: Wed, 25 Jan 2017 22:13:23 -0000


> Has there been any study of how much compressing Hive Parquet tables with=
 snappy reduces storage=C2=A0space or simply the table size in quantitative term=
s?

http://www.slideshare.net/oom65/file-format-benchmarks-avro-json-orc-parque=
t/20

Since SNAPPY is just LZ77, I would assume it would be useful in cases of Pa=
rquet leaves containing text with large common sub-chunks (like URLs or log =
data).

If you want to experiment with that corner case, the L_COMMENT field from T=
PC-H lineitem is a good compression-thrasher.

Cheers,
Gopal