Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Received-SPF: pass (nike.apache.org: domain of stepinto@live.com designates
 65.55.111.103 as permitted sender)
Message-ID: <BLU0-SMTP178F5149FF6DF6F7F362D4AD820@phx.gbl>
MIME-Version: 1.0
In-Reply-To: 
 <CANb5z2K4c=2Wj0y+4z8oupSysr4v-j2GCnuLjLG1R+2B2cUCiw@mail.gmail.com>
References: <BLU0-SMTP166AD025B0B3206D404EB77AD830@phx.gbl>
 <CAH29n6MTo=NoODFpRiMkgyfgRouPg1+qLOZfPQEsaVv=nxnGbA@mail.gmail.com>
 <CAA5C_pu-BVrw08K+MjmgVbZWZ8MVG+LTnBRSC0UxhV5Oe3MfOQ@mail.gmail.com>
 <CAF-WD4T5aJP4_7U8vVzBkZBXQS6-zhZV8y9EF6Y7sMb6EMpv=Q@mail.gmail.com>
 <BLU0-SMTP9142B5F11C8C9FCC6138DFAD830@phx.gbl>
 <CAF-WD4Si7LZgtfy-cfMGKt=yw9YmWiRG0TzZAFyzOM+2nrvOYg@mail.gmail.com>
 <CANb5z2K4c=2Wj0y+4z8oupSysr4v-j2GCnuLjLG1R+2B2cUCiw@mail.gmail.com>
From: Chao Shi <stepinto@live.com>
Date: Fri, 28 Feb 2014 10:26:56 +0800
Subject: Re: Support OutputCommitter?
To: dev@crunch.apache.org
Content-Type: multipart/alternative; boundary="047d7bdc05382b2fbf04f36e2dfa"

--047d7bdc05382b2fbf04f36e2dfa
Content-Type: text/plain; charset="ISO-8859-1"

How about introducinug our own OutputFormat? It can delegate to each
registered OutputCommitter (if any).


2014-02-28 1:28 GMT+08:00 Josh Wills <josh.wills@gmail.com>:

> It's possible to have multiple targets running in one Crunch job; in fact
> it was so common that I switched everything over to the named targets in
> order to simplify the bookkeeping. Every output format can run
> independently of every other output format using the code in CrunchOutputs;
> I think the only reason we default to FileOutputFormat is b/c it's an
> exception for an MR config to _not_ have an OuputFormat configured, even if
> it's never used.
>
>
> On Thu, Feb 27, 2014 at 9:03 AM, Tom White <tom@cloudera.com> wrote:
>
> > Is it possible to have multiple targets that Crunch runs in one
> > MapReduce job? If so then there will be a conflict, and Crunch will
> > need some changes to support this case.
> >
> > Tom
> >
> > On Thu, Feb 27, 2014 at 3:34 PM, Chao Shi <stepinto@live.com> wrote:
> > > Hi Tom,
> > >
> > > I will have to use named-output. About your example DatasetTarget, is
> it
> > > safe to setOutputFormat() explicitly here? I guess this may conflict
> with
> > > other targets that only use the same trick. Is it possible for us to
> > have a
> > > general approach to get OutputCommitter work?
> > > Hi Chao,
> > >
> > > Crunch doesn't call the output committer explicitly itself, it's
> > > called by the MR framework as a normal part of running a job. However,
> > > in Crunch's MapReduceTarget#configureForMapReduce the output format is
> > > not typically set for the named-output case (which is the only case
> > > that is executed now, as I discovered in the thread mentioned below),
> > > so it defaults to FileOutputFormat, with its semantics. (This is why
> > > HBaseTarget calls FileOutputFormat.setOutputPath, which it wouldn't
> > > have to if it set the output format explicitly to HBase's
> > > TableOutputFormat.)
> > >
> > > Are you setting the HCatOutputFormat in the named-output case? In the
> > > Crunch Target I'm writing I've set the OutputFormat explicitly:
> > >
> >
> https://github.com/tomwhite/kite/blob/CDK-308-dataset-output-format/kite-data/kite-data-crunch/src/main/java/org/kitesdk/data/crunch/DatasetTarget.java#L106
> > >
> > > Cheers,
> > > Tom
> > >
> > > On Thu, Feb 27, 2014 at 7:54 AM, Gabriel Reid <gabriel.reid@gmail.com>
> > > wrote:
> > >> For reference, here's the link to the previous thread on this:
> > >>
> > >
> >
> http://mail-archives.apache.org/mod_mbox/crunch-dev/201401.mbox/%3cCAF-WD4Sig2n7yMxiZSji8trQy-8wfUy5_7dnKC=dkSxmrfSPVA@mail.gmail.com%3e
> > >>
> > >> On Thu, Feb 27, 2014 at 7:56 AM, Josh Wills <jwills@cloudera.com>
> > wrote:
> > >>> +tom
> > >>>
> > >>> Didn't Tom have a thing like this a little while ago?
> > >>>
> > >>>
> > >>> On Wed, Feb 26, 2014 at 8:04 PM, Chao Shi <stepinto@live.com> wrote:
> > >>>
> > >>>> Hi crunch devs,
> > >>>>
> > >>>> I'm developing target wrapper for HCatOutputFormat, which uses a
> > custom
> > >>>> OutputCommiter to get results committed to hive. It seems its
> > >>>> OutputCommitter is not called at all. Looking into the code, I can't
> > > find
> > >>>> where crunch calls it. Is it really supported?
> > >>>>
> > >>>> Thanks,
> > >>>> Chao
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Director of Data Science
> > >>> Cloudera <http://www.cloudera.com>
> > >>> Twitter: @josh_wills <http://twitter.com/josh_wills>
> >
>

--047d7bdc05382b2fbf04f36e2dfa--