Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Received-SPF: pass (nike.apache.org: domain of stepinto@live.com designates
 65.55.111.80 as permitted sender)
Message-ID: <BLU0-SMTP9142B5F11C8C9FCC6138DFAD830@phx.gbl>
MIME-Version: 1.0
In-Reply-To: 
 <CAF-WD4T5aJP4_7U8vVzBkZBXQS6-zhZV8y9EF6Y7sMb6EMpv=Q@mail.gmail.com>
References: <BLU0-SMTP166AD025B0B3206D404EB77AD830@phx.gbl>
	<CAH29n6MTo=NoODFpRiMkgyfgRouPg1+qLOZfPQEsaVv=nxnGbA@mail.gmail.com>
	<CAA5C_pu-BVrw08K+MjmgVbZWZ8MVG+LTnBRSC0UxhV5Oe3MfOQ@mail.gmail.com>
	<CAF-WD4T5aJP4_7U8vVzBkZBXQS6-zhZV8y9EF6Y7sMb6EMpv=Q@mail.gmail.com>
Date: Thu, 27 Feb 2014 23:34:18 +0800
Subject: Re: Support OutputCommitter?
From: Chao Shi <stepinto@live.com>
To: dev@crunch.apache.org
Content-Type: multipart/alternative; boundary="f46d04428e7003dc9a04f3650e65"

--f46d04428e7003dc9a04f3650e65
Content-Type: text/plain; charset="ISO-8859-1"

Hi Tom,

I will have to use named-output. About your example DatasetTarget, is it
safe to setOutputFormat() explicitly here? I guess this may conflict with
other targets that only use the same trick. Is it possible for us to have a
general approach to get OutputCommitter work?
Hi Chao,

Crunch doesn't call the output committer explicitly itself, it's
called by the MR framework as a normal part of running a job. However,
in Crunch's MapReduceTarget#configureForMapReduce the output format is
not typically set for the named-output case (which is the only case
that is executed now, as I discovered in the thread mentioned below),
so it defaults to FileOutputFormat, with its semantics. (This is why
HBaseTarget calls FileOutputFormat.setOutputPath, which it wouldn't
have to if it set the output format explicitly to HBase's
TableOutputFormat.)

Are you setting the HCatOutputFormat in the named-output case? In the
Crunch Target I'm writing I've set the OutputFormat explicitly:
https://github.com/tomwhite/kite/blob/CDK-308-dataset-output-format/kite-data/kite-data-crunch/src/main/java/org/kitesdk/data/crunch/DatasetTarget.java#L106

Cheers,
Tom

On Thu, Feb 27, 2014 at 7:54 AM, Gabriel Reid <gabriel.reid@gmail.com>
wrote:
> For reference, here's the link to the previous thread on this:
>
http://mail-archives.apache.org/mod_mbox/crunch-dev/201401.mbox/%3cCAF-WD4Sig2n7yMxiZSji8trQy-8wfUy5_7dnKC=dkSxmrfSPVA@mail.gmail.com%3e
>
> On Thu, Feb 27, 2014 at 7:56 AM, Josh Wills <jwills@cloudera.com> wrote:
>> +tom
>>
>> Didn't Tom have a thing like this a little while ago?
>>
>>
>> On Wed, Feb 26, 2014 at 8:04 PM, Chao Shi <stepinto@live.com> wrote:
>>
>>> Hi crunch devs,
>>>
>>> I'm developing target wrapper for HCatOutputFormat, which uses a custom
>>> OutputCommiter to get results committed to hive. It seems its
>>> OutputCommitter is not called at all. Looking into the code, I can't
find
>>> where crunch calls it. Is it really supported?
>>>
>>> Thanks,
>>> Chao
>>>
>>
>>
>>
>> --
>> Director of Data Science
>> Cloudera <http://www.cloudera.com>
>> Twitter: @josh_wills <http://twitter.com/josh_wills>

--f46d04428e7003dc9a04f3650e65--