hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lefty Leverenz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
Date Tue, 03 Feb 2015 00:54:35 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302595#comment-14302595
] 

Lefty Leverenz commented on HIVE-9500:
--------------------------------------

Doc note:  "hive.serialization.extend.nesting.levels" and "hive.serialization.extend.additional.nesting.levels"
are SerDe properties, not HiveConf properties.  Their documentation should go in the Hive
Types wikidoc, either in a new section for structs or in the existing "Complex Types" section,
and perhaps in the DDL wikidoc.  (One of these days we should add a section on SerDe properties
to the Hive SerDes wikidoc.)

* [Hive Data Types -- Complex Types | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-ComplexTypes]
* [DDL -- Create Table -- Row Format, Storage Format, and SerDe | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RowFormat,StorageFormat,andSerDe]
* [DDL -- Alter Table -- Add SerDe Properties | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AddSerDeProperties]
* [Hive SerDes | https://cwiki.apache.org/confluence/display/Hive/SerDe]

Javadoc descriptions for the properties are in serde/src/java/org/apache/hadoop/hive/serde2/SerDeParameters.java:

{code}
+	/**
+	 * To be backward-compatible, initialize the first 3 separators to 
+   * be the given values. Default number of separators to be 8; If only
+   * hive.serialization.extend.nesting.levels is set, extend the number of
+   * separators to be 24; if hive.serialization.extend.additional.nesting.levels
+   * is set, extend the number of separators to 154.
+	 * @param tbl
+	 */
{code}

Editorial review:  Please align the asterisks and change the first semicolon to a period ("to
be 8; If only...").

> Support nested structs over 24 levels.
> --------------------------------------
>
>                 Key: HIVE-9500
>                 URL: https://issues.apache.org/jira/browse/HIVE-9500
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Aihua Xu
>            Assignee: Aihua Xu
>              Labels: SerDe
>             Fix For: 1.2.0
>
>         Attachments: HIVE-9500.1.patch
>
>
> Customer has deeply nested avro structure and is receiving the following error when performing
queries.
> 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException:
Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level
24
> Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels
is set to true, while the customers have the requirement to support more than that. 
> It would be better to make the supported levels configurable or completely removed (i.e.,
we can support any number of levels). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message