hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sagar Naik <sn...@attributor.com>
Subject Re: explode on array of struct and explode UDTF
Date Fri, 13 May 2011 21:11:35 GMT
Does that mean I need to override UDTFExplode ?
In that case, every nested struct /list  serde needs to implement this ?

Correct if I m wrong



-Sagar
On May 13, 2011, at 6:51 AM, Edward Capriolo wrote:

> You must have a mistake in your object inspector. I am doing something
> similar with nested structs in arrays. This is a struct that has a few
> top level properties, and then one property is a list of structs one
> level of nesting only.
> 
> 	public ObjectInspector initialize(ObjectInspector[] arguments)
> 			throws UDFArgumentException {
> 	ret = new Object[2];
> 		oi=arguments;
> 		
> 		List<String> fieldNames = new ArrayList<String>();
> 		List<ObjectInspector> fieldInspectors = new ArrayList<ObjectInspector>();
> 		
> 		
> 			List<String> structFieldNames = new ArrayList<String>();
> 			List<ObjectInspector> structFieldInspectors = new
> ArrayList<ObjectInspector>();
> 			structFieldNames.add("id");
> 			structFieldNames.add("acount");
> 			structFieldNames.add("bcount");
> 			
> 			structFieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);
> 			structFieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);
> 			structFieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);
> 			
> 			fieldNames.add("redirectList");
> 			fieldInspectors.add(
> 					ObjectInspectorFactory.getStandardListObjectInspector(
> 							ObjectInspectorFactory.getStandardStructObjectInspector(structFieldNames,
> structFieldInspectors)
> 							)
> 			);	
> 		
> 		//sl.getRecordCount()
> 		fieldNames.add("cCount");
> 		fieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);
> 		
> 		StructObjectInspector soi =
> ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames,
> fieldInspectors);
> 		return soi;
> 
> I hope this helps.
> 
> On Thu, May 12, 2011 at 9:45 PM, Sagar Naik <snaik@attributor.com> wrote:
>> Hi ,
>> 
>> I have a four column external table.
>> One of the col is array of struct of five fields.
>> I use a custom serde, to read out hadoop seq file and populate values on to hive
row.
>> The array of struct is ArrayList of Object[5].
>> 
>> All queries run fine.
>> Except the explode query which gives a class cast exception and saying object array
is being casted as string
>> 
>> 
>> I dont knw wht is the issue, but the other serde along same lines works super fine.
>> 
>> pl guide  me
>> 
>> 
>> at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector.getPrimitiveWritableObject(JavaStringObjectInspector.java:35)
>>        at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:206)
>>        at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:480)
>>        at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:426)
>>        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:555)
>>        at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
>>        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
>>        at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
>>        at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
>>        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
>>        at org.apache.hadoop.hive.ql.exec.LateralViewJoinOperator.processOp(LateralViewJoinOperator.java:133)
>>        at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
>>        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
>>        at org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:111)
>>        at org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:44)
>>        at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:82)
>>        at org.apache.hadoop.hive.ql.udf.generic.GenericUDTFExplode.process(GenericUDTFExplode.java:75)
>> 
>> 
>> 
>> -Sagar


Mime
View raw message