hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lukas Nalezenec <lukas.naleze...@firma.seznam.cz>
Subject How to read Protobuffers in Hive
Date Wed, 25 Mar 2015 16:28:55 GMT
Hi,
I am trying to write Serde + ObjectInspectors for reading Protobuffers 
in Hive.
I tried to use class ProtocolBuffersStructObjectInspector from Hive but 
it last worked with old protobuffer version 2.3.
I tried to use ObjectInspector from Twitter Elephant-bird but it does 
not work too.

It looks like that it could help if I havent used DynamicMessage$Builder 
. The problem is that DynamicMessage$Builder cannot be reused.
When message is build from builder the builder field "fields" is set tu 
null and it throws NPE on second build() call.

...
	at com.lukas.AbstractProtobufStructObjectInspector.setStructFieldData_default(AbstractProtobufStructObjectInspector.java:253)
	at com.lukas.ProtobufStructObjectInspector.setStructFieldData(ProtobufStructObjectInspector.java:61)
	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:325)
	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:324)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:630)
	... 9 more
Caused by: java.lang.NullPointerException
	at com.google.protobuf.DynamicMessage$Builder.clearField(DynamicMessage.java:386)
	at com.google.protobuf.DynamicMessage$Builder.clearField(DynamicMessage.java:252)
	at com.lukas.AbstractProtobufStructObjectInspector.setStructFieldData_default(AbstractProtobufStructObjectInspector.java:177)
	... 13 more



I am using Hive 10 but I am also interested in solution for Hive 13.

Does anybody have working Protobuffer Serde ?

Thanks
Best Regards

Lukas

Mime
View raw message