hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Scaffid <sscaff...@tripadvisor.com>
Subject Re: Custom SerDe and getting column comments
Date Fri, 27 Jul 2012 12:47:42 GMT

On Jul 26, 2012, at 9:05 PM, Travis Crawford wrote:

> Currently you can't use comments with custom serde's. Look into
> MetaStoreUtils.getFieldsFromDeserializer and you'll see "from
> deserializer" is hard-coded as the comment.

I found that. I also found in org.apache.hadoop.hive.ql.metadata.Table.getCols() that a decision
is made to get the column info from either the table SD or the SerDe based on the result of
SerDeUtils.shouldGetColsFromSerDe()... But I don't understand why.

Is there some way I can get the comment info from the metadata into the ObjectInspectors used
by my SerDe? Would that be a bad idea?

> This is definitely an area for improvement.
> 
> Is your serde reporting columns, or do you have them stored in the
> metastore? If you "describe extended" your table do you see the
> comments stored in the metastore?

Indeed, the comments show up there, but not in the python script's output. Perhaps I shall
look more closely at that code.

> 
> On Thu, Jul 26, 2012 at 4:17 PM, Stephen Scaffid
> <sscaffidi@tripadvisor.com> wrote:
>> I have a custom SerDe we are using and it works well. However, we have one issue
with it - an application used to allow users to maintain table and column descriptions does
not work with it.
>> 
>> We store the descriptions as "comments" on the tables and columns, via a python script
that simply executes the necessary HQL to set the comment, but then uses the Hive class to
retrieve it from metadata.
>> 
>> The problem is that any comment we try to set ends up getting set to "from deserializer"
on any table using our custom SerDe. This doesn't happen with tables using the LazySimpleSerDe.
>> 
>> Is there a way to make our SerDe work with comments?


Mime
View raw message