Return-Path: X-Original-To: apmail-crunch-dev-archive@www.apache.org Delivered-To: apmail-crunch-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7A53A10D26 for ; Tue, 3 Mar 2015 21:17:08 +0000 (UTC) Received: (qmail 52247 invoked by uid 500); 3 Mar 2015 21:17:05 -0000 Delivered-To: apmail-crunch-dev-archive@crunch.apache.org Received: (qmail 52207 invoked by uid 500); 3 Mar 2015 21:17:05 -0000 Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@crunch.apache.org Delivered-To: mailing list dev@crunch.apache.org Received: (qmail 52194 invoked by uid 500); 3 Mar 2015 21:17:05 -0000 Delivered-To: apmail-incubator-crunch-dev@incubator.apache.org Received: (qmail 52191 invoked by uid 99); 3 Mar 2015 21:17:05 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Mar 2015 21:17:05 +0000 Date: Tue, 3 Mar 2015 21:17:05 +0000 (UTC) From: "Josh Wills (JIRA)" To: crunch-dev@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CRUNCH-502) OutputFormat has inconsistent context state in interface functions MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CRUNCH-502?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Wills updated CRUNCH-502: ------------------------------ Attachment: CRUNCH-502c.patch Managed to simplify this and make it less awful, if still a bit odd. I veri= fied it works on both hadoop1 and hadoop2, will commit it soon. > OutputFormat has inconsistent context state in interface functions > ------------------------------------------------------------------ > > Key: CRUNCH-502 > URL: https://issues.apache.org/jira/browse/CRUNCH-502 > Project: Crunch > Issue Type: Bug > Components: IO > Affects Versions: 0.12.0 > Reporter: M=C4=81rti=C5=86=C5=A1 Kalv=C4=81ns > Assignee: Josh Wills > Attachments: CRUNCH-502.patch, CRUNCH-502b.patch, CRUNCH-502c.pat= ch > > > I created example project to demonstrate problematic behaviour: > https://github.com/sisidra/crunch-ofb > 1. FormatBundle config is not populated to Configuration in checkOutputSp= ecs: > https://github.com/sisidra/crunch-ofb/blob/master/src/main/java/com/spoti= fy/crunch/bugreport/MyOutputFormat.java#L39 > {code} > 15/03/02 15:40:24 INFO bugreport.MyOutputFormat: my.config.key (checkOutp= utSpecs): null > 15/03/02 15:40:24 ERROR bugreport.MyOutputFormat: Wrong my.config.key val= ue in checkOutputSpecs! > {code} > 2. TaskAttemptContext. getTaskAttemptID().toString() is different in getR= ecordWriter and getOutputCommitter: > {code} > 2015-03-02 15:40:38,960 INFO [main] com.spotify.crunch.bugreport.MyOutput= Format: TaskAttemptID (getOutputCommitter): attempt_1422406067005_0121_m_00= 0000_0 > ... > 2015-03-02 15:40:39,789 INFO [main] com.spotify.crunch.bugreport.MyOutput= Format: TaskAttemptID (getRecordWriter): attempt_1422406067005_out0_0121_m_= 000000_0 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)