incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Potter <>
Subject Saving data using Lzo compression using HCatalog from Pig
Date Wed, 30 Jan 2013 21:36:55 GMT
Been struggling with this one for a bit ... Lzo compression is enabled
by default for my Hadoop cluster. If I forget to turn off compression
from my Pig scripts that create data in Hive using HCatalog, then the
partitions get created but I can't read the data back. I don't have
the error handy but it looks like the read-side doesn't treat the data
as compressed.

So I've resorted to adding the following to my scripts:

SET mapreduce.output.compress false;
SET mapred.output.compress false;
SET output.compression.enabled false;

One of the those seems to do the trick ;-)

I'd really like to store my Hive data compressed but haven't figured
out how to enable this with HCatalog. Seems like it's either not
supported yet or I'm missing something simple in my HQL DDL table


View raw message