Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Received-SPF: pass (athena.apache.org: domain of mkwhit@gmail.com designates
 209.85.223.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAF-WD4Sig2n7yMxiZSji8trQy-8wfUy5_7dnKC=dkSxmrfSPVA@mail.gmail.com>
References: 
 <CAF-WD4Sig2n7yMxiZSji8trQy-8wfUy5_7dnKC=dkSxmrfSPVA@mail.gmail.com>
From: Micah Whitacre <mkwhit@gmail.com>
Date: Wed, 29 Jan 2014 15:21:40 -0600
Message-ID: 
 <CAKFfB9miWbmYv7+H+U4TfD49ifeDuGFmyBTfZMo1TeYH37DTfQ@mail.gmail.com>
Subject: Re: Output Committers and Crunch Targets
To: dev@crunch.apache.org
Content-Type: multipart/alternative; boundary=001a11346b12072cf404f122887b

--001a11346b12072cf404f122887b
Content-Type: text/plain; charset=ISO-8859-1

>> I would expect that
>> named outputs would not be used in my simple pipeline, so name would
>> be null, but it actually seems that the name parameter is 'out0'. So
>> my first question is: what determines when named outputs are used?

Looking at the code the output is always named[1] regardless of the number
of outputs.  Do you believe the use of a name is causing an issue with the
utilization of your custom committer?

Regarding your second question I need to do a bit more digging to answer
for certain.

[1] -
https://github.com/apache/crunch/blob/master/crunch-core/src/main/java/org/apache/crunch/impl/mr/plan/MSCROutputHandler.java#L64


On Wed, Jan 29, 2014 at 10:11 AM, Tom White <tom@cloudera.com> wrote:

> Hi,
>
> I'm writing a Crunch Target that is a MapReduceTarget, but not a
> PathTarget, since it writes to files in a partitioned manner, so there
> is not necessarily a single output path. I'm confused about the 'name'
> parameter in configureForMapReduce() though - I would expect that
> named outputs would not be used in my simple pipeline, so name would
> be null, but it actually seems that the name parameter is 'out0'. So
> my first question is: what determines when named outputs are used?
>
> In the past this hasn't been a problem (e.g. with the Parquet target),
> but this output format has a custom output committer which isn't being
> used. Instead it looks like the default file committer is being used
> by Crunch, so the job fails. Is it possible to use custom output
> committers with Crunch?
>
> My code is here:
>
> https://github.com/tomwhite/kite/blob/CDK-251-mr/kite-data/kite-data-crunch/src/main/java/org/kitesdk/data/crunch/DatasetTarget.java#L100
>
> Cheers,
> Tom
>

--001a11346b12072cf404f122887b--