Return-Path: X-Original-To: apmail-crunch-dev-archive@www.apache.org Delivered-To: apmail-crunch-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9C37C102ED for ; Wed, 26 Mar 2014 03:11:20 +0000 (UTC) Received: (qmail 41872 invoked by uid 500); 26 Mar 2014 03:11:20 -0000 Delivered-To: apmail-crunch-dev-archive@crunch.apache.org Received: (qmail 41742 invoked by uid 500); 26 Mar 2014 03:11:18 -0000 Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@crunch.apache.org Delivered-To: mailing list dev@crunch.apache.org Received: (qmail 41732 invoked by uid 500); 26 Mar 2014 03:11:17 -0000 Delivered-To: apmail-incubator-crunch-dev@incubator.apache.org Received: (qmail 41727 invoked by uid 99); 26 Mar 2014 03:11:17 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Mar 2014 03:11:17 +0000 Date: Wed, 26 Mar 2014 03:11:17 +0000 (UTC) From: "Micah Whitacre (JIRA)" To: crunch-dev@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CRUNCH-370) Update Parquet dependency in Crunch pom MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CRUNCH-370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Micah Whitacre updated CRUNCH-370: ---------------------------------- Attachment: CRUNCH-370.patch Patch upgrading master to 1.3.2. The build passed successfully. I haven't dug in though to see if Parquet made any non-passive changes between versions that would make this upgrade not desirable. [~tomwhite], since you wrote some of the original Parquet support do you have any objections or concerns with making this upgrade? > Update Parquet dependency in Crunch pom > --------------------------------------- > > Key: CRUNCH-370 > URL: https://issues.apache.org/jira/browse/CRUNCH-370 > Project: Crunch > Issue Type: Improvement > Components: IO > Affects Versions: 0.9.0 > Reporter: Anandsagar Kothapalli > Assignee: Micah Whitacre > Attachments: CRUNCH-370.patch > > > Currently crunch is supporting avro to parquet conversion using AvroParquetFileTarget, AvroParquetFileSource classes. When I used these classes to convert avro to parquet files, I got the following exception in some cases: "org.apache.crunch.CrunchRuntimeException: parquet.io.ParquetEncodingException: empty fields are illegal, the field should be ommited completely instead" > After further debugging I found out that this issue is related to AvroWriteSupport class in parquet, which was fixed as part of milestone 1.2.3 https://github.com/Parquet/parquet-mr/issues/162. Latest parquet version is 1.3.2. > But crunch is still using parquet 1.2.0 https://github.com/apache/crunch/blob/master/pom.xml#L77 > As part of this improvement, parquet dependency version in crunch will be updated if not to latest then at least to 1.2.3 -- This message was sent by Atlassian JIRA (v6.2#6252)