Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C46E510EF2 for ; Mon, 6 May 2013 07:48:57 +0000 (UTC) Received: (qmail 94184 invoked by uid 500); 6 May 2013 07:48:53 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 93966 invoked by uid 500); 6 May 2013 07:48:52 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 93950 invoked by uid 99); 6 May 2013 07:48:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 May 2013 07:48:52 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rahul.rec.dgp@gmail.com designates 209.85.128.170 as permitted sender) Received: from [209.85.128.170] (HELO mail-ve0-f170.google.com) (209.85.128.170) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 May 2013 07:48:46 +0000 Received: by mail-ve0-f170.google.com with SMTP id 15so2988854vea.1 for ; Mon, 06 May 2013 00:48:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=fk/eJH2d2RnTA2l69n/oaeWz9os3V5cg47MlZD6Q1Mg=; b=cIhJXTo11h6DjfDKw/jYZlBTf8Og1+RtaCeilK3ZJovx97legi5MBWKjwDoOmNkMbO GuCCA5ZNAQJ6QHXbbuGyM1W9Xs05XqDAA3xoZY87i4H1elVGUungHdap+dYWutLZth0C pp+OTxEPgfBb+2hsVl7JjVAsDk7Z6ddsa2gchcSNc8Rd49gpLb4b6iH0pha6VtYRFXg/ h0GwXGA6/Yda9Gx8bfQ6pU08hjR4WiWTPnVElnxBmBAtiuIe/j3KcRmk6DxX+VoTP1gB Lui3ruKwKOoxtjmMe5rlEythqOYrsSS+GT4wL0MSnfFxtUVNmpuhD96TBDMBjEybwhaN Sp5w== X-Received: by 10.52.35.196 with SMTP id k4mr5493449vdj.77.1367826505856; Mon, 06 May 2013 00:48:25 -0700 (PDT) MIME-Version: 1.0 Received: by 10.59.1.170 with HTTP; Mon, 6 May 2013 00:48:05 -0700 (PDT) In-Reply-To: References: From: Rahul Bhattacharjee Date: Mon, 6 May 2013 13:18:05 +0530 Message-ID: Subject: Re: no _SUCCESS file in MR output directory. To: "user@hadoop.apache.org" , user@oozie.apache.org Content-Type: multipart/alternative; boundary=20cf3079be3cff011904dc07ec3a X-Virus-Checked: Checked by ClamAV on apache.org --20cf3079be3cff011904dc07ec3a Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I wanted to confirm whether oozie disables the _SUCCESS file creation when it triggers a MR job. I am triggering a MR job (actually a bunch of 'em) from oozie and the workflow completes successfully , however I do not see any kind of _SUCCESS file in the output directory. When I set the file output committer's configuration (mapreduce.fileoutputcommitter. marksuccessfuljobs ) to true.It generates the success file. Wanted to confirm if oozie does the disabling of success file creation. =E2=80=8BThanks, Rahul=E2=80=8B On Mon, May 6, 2013 at 12:34 PM, Rahul Bhattacharjee < rahul.rec.dgp@gmail.com> wrote: > Oozie is being used for triggering the MR job. Looks like oozie disables > the success file creation using the configuration that you have mentioned > for FileOutputCommitter. > > I have enabled it by setting this property in conf. > > Rahul > > > On Mon, May 6, 2013 at 9:38 AM, Rahul Bhattacharjee < > rahul.rec.dgp@gmail.com> wrote: > >> Thanks Harsh for the pointers. I will find out more on this. >> >> >> On Sun, May 5, 2013 at 11:26 PM, Harsh J wrote: >> >>> I can think of a few, most obvious ones: >>> >>> 1. Job didn't succeed and/or the file was deleted (*shields self*) >>> 2. Job overrode the default FileOutputCommitter with something that >>> doesn't do success marking. >>> 3. Job specifically asked to not create such files, via config >>> mapreduce.fileoutputcommitter.marksuccessfuljobs or so, set to false. >>> >>> On Sun, May 5, 2013 at 9:54 PM, Rahul Bhattacharjee >>> wrote: >>> > Hi, >>> > >>> > >>> > A few days back , I was going through a MR job's output , but there >>> wasn't >>> > any _SUCCESS file in the output directory. >>> > I was wondering what all reasons for this (no _SUCCESS file)? >>> > >>> > Thanks, >>> > Rahul >>> >>> >>> >>> -- >>> Harsh J >>> >> >> > --20cf3079be3cff011904dc07ec3a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I wanted to confirm whether oozie disables = the _SUCCESS file creation when it triggers a MR job.

I am triggering a MR job (actually a bunch of 'em) from oozie and the w= orkflow completes successfully , however I do not see any kind of _SUCCESS = file in the output directory.
When I set the file output committer's configuration (mapreduce.fileout= putcommitter.
marksuccessfuljobs
) to true.It generate= s the success file. Wanted to confirm if oozie does the disabling of succes= s file creation.

=E2=80=8BThanks,
Rah= ul=E2=80=8B


On Mon, May 6, 2013 at = 12:34 PM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com> wrote:
Oozie is being used for triggering= the MR job. Looks like oozie disables the success file creation using the = configuration that you have mentioned for FileOutputCommitter.

I have enabled i= t by setting this property in conf.

Rahul


On Mon, May 6, 2013= at 9:38 AM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com> wrote:
Thanks Harsh for the pointers. I w= ill find out more on this.


On Sun, May 5, 2013 at 11:26 PM, Harsh J <harsh@cloudera.com> wrote:
I can think of a few, most obvious ones:

1. Job didn't succeed and/or the file was deleted (*shields self*)
2. Job overrode the default FileOutputCommitter with something that
doesn't do success marking.
3. Job specifically asked to not create such files, via config
mapreduce.fileoutputcommitter.marksuccessfuljobs or so, set to false.

On Sun, May 5, 2013 at 9:54 PM, Rahul Bhattacharjee
<rahul.rec.= dgp@gmail.com> wrote:
> Hi,
>
>
> A few days back , I was going through a MR job's output , but ther= e wasn't
> any _SUCCESS file in the output directory.
> I was wondering what all reasons for this =C2=A0(no _SUCCESS file)? >
> Thanks,
> Rahul



--
Harsh J



--20cf3079be3cff011904dc07ec3a--