hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Marron <Peter.Mar...@trilliumsoftware.com>
Subject Table Wrapper
Date Tue, 25 Jun 2013 14:29:45 GMT
Hi,

Running Hive 0.11.0 over Hadoop 1.0.4.

I would like to be able to "wrap" a Hive table.

So, if I have table "X" which uses SerDe "s" and InputFormat "i"
then I would like to be able to create a table "Y" which has a
SerDe "ws" which is a wrapper of "s" (and so can encapsulate an instance of "s")
and an InputFormat "wi" which is a wrapper of "I" (and similarly encapsulates an
instance of "i").  So far I have done this by creating a table like this

CREATE TABLE Y (... copy of underlying table's columns...)
ROW FORMAT SERDE 'ws'
WITH SERDEPROPERTIES (...
'wrapped.serde.name'='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe',
'wrapped.inputformat.name'='TextInputFormat',
'serialization.format'='|',              'field.delim'='|'
)
STORED AS
  INPUTFORMAT 'wi'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
TBLPROPERTIES (...);

I have to add the names of the underlying classes "s" and "I"
into the table properties so that I know what to instantiate.
I also have to replicate all the column details of the wrapped table
to ensure the correct information is passed down to the underlying SerDe
when I instantiate it. I also have to know the output format. I have to
explicitly add the default SerDe properties to get it to work.
I have to explicitly provide the default output format too.
If any changes are made to the underlying table then I need to reflect
those changes in my "wrapper" table.
It's a mess.

What I'd like to be able to do is to just parameterise my wrapper
table with the name of the underlying table and using that name
be able to instantiate the correct SerDe and InputFormat.

Is there an easier way to do this?

Any pointers appreciated.

Z


Mime
View raw message