hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher, Pat" <>
Subject RE: Custom SerDe Question
Date Fri, 28 Jan 2011 22:34:57 GMT
Not sure what I did wrong the first time but I tried to create a table with stored type of
textfile and using my custom serde so it had a format line of:

  ROW FORMAT SERDE 'org.myorg.hadoop.hive.udf.MySerDe' STORED AS textfile

Then I loaded a gzipped file using LOAD DATA LOCAL INPATH 'path.gz' INTO TABLE mytable and
it worked as expected, ie the file was read and I'm able to query it using hive.

Sorry to bother and thanks a bunch for the help!  Forcing me to go read more about InputFormats
is a long term help anyway.


From: phil young []
Sent: Friday, January 28, 2011 1:54 PM
Subject: Re: Custom SerDe Question

To be clear, you would then create the table with the clause:

  INPUTFORMAT 'your.custom.input.format'

If you make an external table, you'll then be able to point to a directory (or file) that
contains gzipped files, or uncompressed files.

On Fri, Jan 28, 2011 at 4:52 PM, phil young <<>>
This can be accomplished with a custom input format.

Here's a snippet of the relevant code in the customer RecordReader

            compressionCodecs = new CompressionCodecFactory(jobConf);

            Path file = split.getPath();

            final CompressionCodec codec = compressionCodecs.getCodec(file);

            // open the file and seek to the start of the split

            start = split.getStart();

            end = start + split.getLength();


            FileSystem fs = file.getFileSystem(jobConf);

            fsdat =;


            if (codec != null)


                fsin = codec.createInputStream(fsdat);




                fsin = fsdat;


On Fri, Jan 28, 2011 at 1:57 PM, Christopher, Pat <<>>
I've written a SerDe and I'd like it to be able handle compressed data (gzip).  Hadoop detects
and decompresses on the fly so if you have a compressed data set and you don't need to perform
any custom interpretation of it as you go, hadoop and hive will handle it.  Is there a way
to get Hive to notice the data is compressed, decompress it then push it through the custom
SerDe?  Or will I have to either
  a. add some decompression logic to my SerDe (possibly impossible)
  b. decompress the data before pushing it into a table with my SerDe



View raw message