crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anandsagar Kothapalli (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CRUNCH-370) Update Parquet dependency in Crunch pom
Date Tue, 25 Mar 2014 14:37:16 GMT
Anandsagar Kothapalli created CRUNCH-370:
--------------------------------------------

             Summary: Update Parquet dependency in Crunch pom
                 Key: CRUNCH-370
                 URL: https://issues.apache.org/jira/browse/CRUNCH-370
             Project: Crunch
          Issue Type: Improvement
          Components: MapReduce Patterns
    Affects Versions: 0.9.0
            Reporter: Anandsagar Kothapalli


Currently crunch is supporting avro to parquet conversion using AvroParquetFileTarget, AvroParquetFileSource
classes. When I used these classes to convert avro to parquet files, I got the following exception
in some cases: "org.apache.crunch.CrunchRuntimeException: parquet.io.ParquetEncodingException:
empty fields are illegal, the field should be ommited completely instead"

After further debugging I found out that this issue is related to AvroWriteSupport class in
parquet, which was fixed as part of milestone 1.2.3 https://github.com/Parquet/parquet-mr/issues/162.
Latest parquet version is 1.3.2.

But crunch is still using parquet 1.2.0 https://github.com/apache/crunch/blob/master/pom.xml#L77

As part of this improvement, parquet dependency version in crunch will be updated if not to
latest then at least to 1.2.3 




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message