spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng Lian (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-10136) Parquet support fail to decode Avro arrays of primitive array (e.g. array<array<int>>)
Date Thu, 20 Aug 2015 14:51:47 GMT
Cheng Lian created SPARK-10136:
----------------------------------

             Summary: Parquet support fail to decode Avro arrays of primitive array (e.g.
array<array<int>>)
                 Key: SPARK-10136
                 URL: https://issues.apache.org/jira/browse/SPARK-10136
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.5.0
            Reporter: Cheng Lian
            Assignee: Cheng Lian


The following Avro schema
{noformat}
    record AvroNonNullableArrays {
        array<array<int>> int_arrays_column;
    }
{noformat}
is translated into the following Parquet schema by parquet-avro 1.7.0:
{noformat}
message root {
  required group int_arrays_column (LIST) {
    repeated group array (LIST) {
      repeated int32 array;
    }
  }
}
{noformat}
When making converters, the inner most {{array}} field is covered by {{RepeatedPrimitiveConverter}},
which is introduced in SPARK-9340 and [PR #8070|https://github.com/apache/spark/pull/8070].
However, PR #8070 assumes that repeated converters can only be created for struct fields,
and doesn't take array elements into account.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message