avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Kleppmann <mar...@rapportive.com>
Subject Re: Avro with Snappy compression on Hive
Date Thu, 25 Apr 2013 14:19:51 GMT
I've never Avro output with Hive, but just as a guess, try this:

SET avro.output.codec=snappy;

The mapred.output.compression.codec and mapred.output.compression.type
options are probably redundant.


On 25 April 2013 07:12, nir_zamir <nir.zamir@gmail.com> wrote:

> Hi,
>
> I have a Hive table created with the Avro Serde.
>
> When I add some data to it using the Snappy compression, it still looks
> compressed with deflate (the file starts with
> 'Obj...avro.codec.deflate.avro.Schema' where for raw data compressed with
> Snappy, the Snappy coded is specified at the beginning of the file).
>
> Anything I'm doing wrong?
>
> Here's what I do:
>
> CREATE TABLE p2c_comp_avro
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED as INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   TBLPROPERTIES (
>
>
> 'avro.schema.url'='file:///home/cloudera/bigdata/path_to_conversions_raw.avsc');
>
> SET hive.exec.compress.output=true;
> SET
> mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compression.type=BLOCK;
>
> INSERT OVERWRITE TABLE p2c_comp_avro SELECT * FROM p2c;
>
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-avro.679487.n3.nabble.com/Avro-with-Snappy-compression-on-Hive-tp4027079.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
>

Mime
View raw message