hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <>
Subject SerDe loading external scheme
Date Thu, 05 Apr 2012 16:26:25 GMT
I am working on a hive SerDe where both SerDe and RecordReader need to have
access to an external resource with information.
This external resource could be on hdfs, in hbase, or on a http server.
This situation is very similar to what haivvreo does.

The way i go about it right now is that i store the uri for the external
resource in the SERDEPROPERTIES and then both SerDe and RecordReader use
that to load the resource. I had to jump through some hoops to retrieve the
Properties object (the SERDEPROPERTIES) in the RecordReader, but now it
works. However this is far from optimal, since on a large cluster this
leads to a lot of read request on the external resource.

Since SerDe gets called at least once on the client before the mapreduce
job is started, i would like to load my external resource there, and then
stuff it in the Configuration object, the Properties object or in the
Distributed Cache. Then the SerDes and RecordReaders on the cluster could
get it from there and wouldn't have to access the external resource.

I made the changes. But whatever modification i make to Configuration
object or Properties object on the client in SerDe doesn't make it to the
cluster! Is there a way to do this?

View raw message