crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CRUNCH-481) Support independent output committers for multiple outputs
Date Tue, 03 Feb 2015 11:23:34 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tom White updated CRUNCH-481:
-----------------------------
    Attachment: CRUNCH-481.patch

The test hang is happening in Kite (see https://issues.cloudera.org/browse/CDK-756, you have
to apply the patch there and adjust the Crunch version in the POM).

Here's an updated patch which fixes the problem by setting the job ID. It's slightly tricky
due to Hadoop 1/2 differences. It passes all Crunch tests when run under both Hadoop 1 and
2, and passes the test in CDK-756.

> Support independent output committers for multiple outputs
> ----------------------------------------------------------
>
>                 Key: CRUNCH-481
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-481
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>            Reporter: Aniket Kulkarni
>            Assignee: Josh Wills
>            Priority: Minor
>             Fix For: 0.12.0
>
>         Attachments: CRUNCH-481.patch, CRUNCH-481.patch, CRUNCH-481.patch, CRUNCH-481c.patch
>
>
> I faced this issue while trying to write to Kite and HDFS in the same pipeline. A similar
issue was logged for Kite[1][2]. 
> I was attempting to write a PCollection to Kite and a different PTable to HDFS as a text
file. The write to Kite succeeded, however the write to HDFS only produced a _SUCCESS file
with no text file.
> Commenting out the write to Kite seems to solve the issue and I can see the text file
being written.
> [1] - https://issues.cloudera.org/browse/CDK-756
> [2] - http://mail-archives.apache.org/mod_mbox/crunch-dev/201401.mbox/%3CCAF-WD4QCUe0Toh3qewpDNnom3u786PVJLgH7T6Go_AbcTpLTaw@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message