Return-Path: 
 <dev-return-63793-apmail-hive-dev-archive=hive.apache.org@hive.apache.org>
X-Original-To: apmail-hive-dev-archive@www.apache.org
Delivered-To: apmail-hive-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7501F10E22
	for <apmail-hive-dev-archive@www.apache.org>;
 Thu, 19 Sep 2013 13:39:56 +0000 (UTC)
Received: (qmail 10122 invoked by uid 500); 19 Sep 2013 13:39:55 -0000
Delivered-To: apmail-hive-dev-archive@hive.apache.org
Received: (qmail 10027 invoked by uid 500); 19 Sep 2013 13:39:54 -0000
Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@hive.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@hive.apache.org>
List-Post: <mailto:dev@hive.apache.org>
List-Id: <dev.hive.apache.org>
Reply-To: dev@hive.apache.org
Delivered-To: mailing list dev@hive.apache.org
Received: (qmail 9774 invoked by uid 500); 19 Sep 2013 13:39:54 -0000
Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org
Received: (qmail 9699 invoked by uid 99); 19 Sep 2013 13:39:52 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Sep 2013 13:39:52 +0000
Date: Thu, 19 Sep 2013 13:39:52 +0000 (UTC)
From: "Chaoyu Tang (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12669458.1379597938065.161975.1379597992509@arcas>
In-Reply-To: <JIRA.12669458.1379597938065@arcas>
References: <JIRA.12669458.1379597938065@arcas>
Subject: [jira] [Created] (HIVE-5320) Querying a table with nested struct
 type over JSON data results in errors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Chaoyu Tang created HIVE-5320:
---------------------------------

             Summary: Querying a table with nested struct type over JSON data results in errors
                 Key: HIVE-5320
                 URL: https://issues.apache.org/jira/browse/HIVE-5320
             Project: Hive
          Issue Type: Bug
          Components: Serializers/Deserializers
    Affects Versions: 0.9.0
            Reporter: Chaoyu Tang


Querying a table with nested_struct datatype like
==
create table nest_struct_tbl (col1 string, col2 array<struct<a1:string, a2:array<struct<b1:int, b2:string, b3:string>>>>) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'; 
==
over JSON data cause errors including java.lang.IndexOutOfBoundsException or corrupted data. 
The JsonSerDe used is json-serde-1.1.4.jar/json-serde-1.1.4-jar-dependencies.jar.

The cause is that the method:
public List<Object> getStructFieldsDataAsList(Object o) 
in JsonStructObjectInspector.java 
returns a list referencing to a static arraylist "values"
So the local variable 'list' in method serialize of Hive LazySimpleSerDe class is returned with same reference in its recursive calls and its element values are kept on being overwritten in the case STRUCT.

Solutions:
1. Fix in JsonSerDe, and change the field 'values' in java.org.openx.data.jsonserde.objectinspector.JsonStructObjectInspector.java
to instance scope.
Filed a ticket to JSonSerDe (https://github.com/rcongiu/Hive-JSON-Serde/issues/31)
2. Ideally, in the method serialize of class LazySimpleSerDe, we should defensively save a copy of a list resulted from list = soi.getStructFieldsDataAsList(obj) in which case the soi is the instance of JsonStructObjectInspector, so that the recursive calls of serialize can work properly regardless of the extended SerDe implementation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira