Return-Path: 
 <issues-return-64677-apmail-spark-issues-archive=spark.apache.org@spark.apache.org>
X-Original-To: apmail-spark-issues-archive@minotaur.apache.org
Delivered-To: apmail-spark-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5CB6B18B49
	for <apmail-spark-issues-archive@minotaur.apache.org>;
 Thu, 20 Aug 2015 14:51:49 +0000 (UTC)
Received: (qmail 73588 invoked by uid 500); 20 Aug 2015 14:51:47 -0000
Delivered-To: apmail-spark-issues-archive@spark.apache.org
Received: (qmail 73462 invoked by uid 500); 20 Aug 2015 14:51:47 -0000
Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@spark.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@spark.apache.org>
List-Post: <mailto:issues@spark.apache.org>
List-Id: <issues.spark.apache.org>
Delivered-To: mailing list issues@spark.apache.org
Received: (qmail 73264 invoked by uid 99); 20 Aug 2015 14:51:47 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2015 14:51:47 +0000
Date: Thu, 20 Aug 2015 14:51:47 +0000 (UTC)
From: "Cheng Lian (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12857551.1440082259000.119985.1440082307389@Atlassian.JIRA>
In-Reply-To: <JIRA.12857551.1440082259000@Atlassian.JIRA>
References: <JIRA.12857551.1440082259000@Atlassian.JIRA>
 <JIRA.12857551.1440082259644@arcas>
Subject: [jira] [Created] (SPARK-10136) Parquet support fail to decode Avro
 arrays of primitive array (e.g. array<array<int>>)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Cheng Lian created SPARK-10136:
----------------------------------

             Summary: Parquet support fail to decode Avro arrays of primitive array (e.g. array<array<int>>)
                 Key: SPARK-10136
                 URL: https://issues.apache.org/jira/browse/SPARK-10136
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.5.0
            Reporter: Cheng Lian
            Assignee: Cheng Lian


The following Avro schema
{noformat}
    record AvroNonNullableArrays {
        array<array<int>> int_arrays_column;
    }
{noformat}
is translated into the following Parquet schema by parquet-avro 1.7.0:
{noformat}
message root {
  required group int_arrays_column (LIST) {
    repeated group array (LIST) {
      repeated int32 array;
    }
  }
}
{noformat}
When making converters, the inner most {{array}} field is covered by {{RepeatedPrimitiveConverter}}, which is introduced in SPARK-9340 and [PR #8070|https://github.com/apache/spark/pull/8070]. However, PR #8070 assumes that repeated converters can only be created for struct fields, and doesn't take array elements into account.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org