hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yongqiang he <heyongqiang...@gmail.com>
Subject Re: Building Custom RCFiles
Date Thu, 17 Mar 2011 23:04:50 GMT
You need to customize Hive's ColumnarSerde (maybe functions in
LazySerde)'s serde and deserialize function (depends you want to read
or write.). And the main thing is that you need to use your own type
def (not LazyInt/LazyLong).

If your type is int or long (not double/float), casting it to string
only wastes some CPU, but can save you more spaces.

Thanks
Yongqiang
On Thu, Mar 17, 2011 at 3:48 PM, Severance, Steve <sseverance@ebay.com> wrote:
> Hi,
>
>
>
> I am working on building a MR job that generates RCFiles that will become
> partitions of a hive table. I have most of it working however only strings
> (Text) are being deserialized inside of Hive. The hive table is specified to
> use a columnarserde which I thought should allow the writable types stored
> in the RCFile to be deserialized properly.
>
>
>
> Currently all numeric types (IntWritable and LongWritable) come back a null.
>
>
>
> Has anyone else seen anything like this or have any ideas? I would rather
> not convert all my data to strings to use RCFile.
>
>
>
> Thanks.
>
>
>
> Steve

Mime
View raw message