hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aihua Xu (JIRA)" <>
Subject [jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
Date Fri, 30 Jan 2015 18:41:34 GMT


Aihua Xu commented on HIVE-9500:

[~thejas] There are some formats like MultiDelimitSerDe use the functionalities from LazySimpleSerde,
so they will have same issue. 

Actually I do have the challenge to handle escaped delimiter. Originally I thought during
the construction of object inspectors, I would know how many delimiters I need and they will
need to be escaped during serialization and deserialization later. But  serialization function
will pass in an object inspector along with the data (that I'm not familiar with). Any idea
on that?

We could take a simple approach, to use [1, 31] and [-128, -1] as the hardcoded list of delimiters
and escape them.  

> Support nested structs over 24 levels.
> --------------------------------------
>                 Key: HIVE-9500
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Aihua Xu
>            Assignee: Aihua Xu
>              Labels: SerDe
> Customer has deeply nested avro structure and is receiving the following error when performing
> 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException:
Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level
> Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels
is set to true, while the customers have the requirement to support more than that. 
> It would be better to make the supported levels configurable or completely removed (i.e.,
we can support any number of levels). 

This message was sent by Atlassian JIRA

View raw message