hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kulkarni.swarnim@gmail.com" <kulkarni.swar...@gmail.com>
Subject Separators in struct
Date Wed, 11 Jul 2012 21:10:05 GMT
Hello,

I am not sure I understand the significance of separators very well in case
of structs. For instance, for deeply nested structs I usually hit this
exception:

java.lang.ArrayIndexOutOfBoundsException: 9
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)

Digging deeper into the code I found that the size of separators have been
hard-coded to be an array of size "8" with a comment on it.

// Read the separators: We use 8 levels of separators by default, but we should
change this when we allow users to specify more than 10 levels of
separators through DDL.

serdeParams.separators = new byte[8];


If someone can explain this to me, I would really appreciate that. Also is
there a way to change the number of separators so that this exception is
not thrown?

Thanks,

-- 
Swarnim

Mime
View raw message