crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anandsagar Kothapalli (JIRA)" <>
Subject [jira] [Created] (CRUNCH-370) Update Parquet dependency in Crunch pom
Date Tue, 25 Mar 2014 14:37:16 GMT
Anandsagar Kothapalli created CRUNCH-370:

             Summary: Update Parquet dependency in Crunch pom
                 Key: CRUNCH-370
             Project: Crunch
          Issue Type: Improvement
          Components: MapReduce Patterns
    Affects Versions: 0.9.0
            Reporter: Anandsagar Kothapalli

Currently crunch is supporting avro to parquet conversion using AvroParquetFileTarget, AvroParquetFileSource
classes. When I used these classes to convert avro to parquet files, I got the following exception
in some cases: "org.apache.crunch.CrunchRuntimeException:
empty fields are illegal, the field should be ommited completely instead"

After further debugging I found out that this issue is related to AvroWriteSupport class in
parquet, which was fixed as part of milestone 1.2.3
Latest parquet version is 1.3.2.

But crunch is still using parquet 1.2.0

As part of this improvement, parquet dependency version in crunch will be updated if not to
latest then at least to 1.2.3 

This message was sent by Atlassian JIRA

View raw message