avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nir_zamir <nir.za...@gmail.com>
Subject Avro with Snappy compression on Hive
Date Thu, 25 Apr 2013 14:12:44 GMT
Hi,

I have a Hive table created with the Avro Serde.

When I add some data to it using the Snappy compression, it still looks
compressed with deflate (the file starts with
'Obj...avro.codec.deflate.avro.Schema' where for raw data compressed with
Snappy, the Snappy coded is specified at the beginning of the file). 

Anything I'm doing wrong?

Here's what I do:

CREATE TABLE p2c_comp_avro
  ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
  STORED as INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
  OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
  TBLPROPERTIES (
   
'avro.schema.url'='file:///home/cloudera/bigdata/path_to_conversions_raw.avsc');

SET hive.exec.compress.output=true;
SET
mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compression.type=BLOCK;

INSERT OVERWRITE TABLE p2c_comp_avro SELECT * FROM p2c;


Thanks!



--
View this message in context: http://apache-avro.679487.n3.nabble.com/Avro-with-Snappy-compression-on-Hive-tp4027079.html
Sent from the Avro - Users mailing list archive at Nabble.com.

Mime
View raw message