hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Lau <r...@ooyala.com>
Subject Re: ArrayWritableGroupConverter
Date Thu, 14 Aug 2014 17:43:18 GMT
Original Thrift schema:

struct teststruct {
  1: optional string field1;
  2: optional string field2;
  3: optional string field3;
}

struct mainstruct {
  1: optional list<teststruct> teststructs;
}

This parquet file schema was generated:

message ParquetSchema {
  optional group teststructs {
    repeated group teststruct_tuple {
      optional binary field1;
      optional binary field2;
      optional binary field3;
    }
  }
}

When i try to run queries involving this 'teststructs' column, I get this
error:

Failed with exception java.io.IOException:java.lang.RuntimeException:
Invalid parquet hive schema: repeated group teststruct_tuple {
      optional binary field1;
      optional binary field2;
      optional binary field3;
    }



On Thu, Aug 14, 2014 at 8:35 AM, Brock Noland <brock@cloudera.com> wrote:

> Hi,
>
> Can you share your parquet schema?
>
> Brock
>
>
> On Tue, Aug 12, 2014 at 5:06 PM, Raymond Lau <raymond.lau.ca@gmail.com>
> wrote:
>
>> Hello.  (First off, sorry if I accidentally posted to the wrong mailing
>> list before - dev - and you are getting this again)
>>
>> Regarding the ArrayWritableGroupConverter class: I was just wondering how
>> come the field count has to be either 1 or 2?  I'm trying to read a column
>> where the amount is fields is 3 and I'm getting an invalid parquet hive
>> schema (in hive 0.12) error when I try to do so.  It looks like it links
>> back to here.
>>
>> *https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java
>> <https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java>*
>>
>>
>> Thanks,
>> -Raymond
>>
>
>


-- 
*Raymond Lau*
Software Engineer - Intern |
rlau@ooyala.com | (925) 395-3806

Mime
View raw message