hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <>
Subject [jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
Date Fri, 30 Jan 2015 17:18:35 GMT


Thejas M Nair commented on HIVE-9500:

[~aihuaxu] OK, I didn't realize LazySimpleSerde is used for Avro. But wouldn't this limitation
apply only to formats that use LazySimpleSerde. Does it apply to ORC for example (or any format
that does not use LazySimpleSerde) ? I am not sure if we use LazySimpleSerde for serialization
of intermediate results anymore.

Regarding dynamically generating the separators for LazySimpleSerde, you need to ensure that
the delimiter chars are not present in the data, or that they are escaped. How would you handle
that ?

> Support nested structs over 24 levels.
> --------------------------------------
>                 Key: HIVE-9500
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Aihua Xu
>            Assignee: Aihua Xu
>              Labels: SerDe
> Customer has deeply nested avro structure and is receiving the following error when performing
> 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException:
Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level
> Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels
is set to true, while the customers have the requirement to support more than that. 
> It would be better to make the supported levels configurable or completely removed (i.e.,
we can support any number of levels). 

This message was sent by Atlassian JIRA

View raw message