Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.223.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1367850274.24846.YahooMailNeo@web125601.mail.ne1.yahoo.com>
References: 
 <CAO7hTbMM+3s49Ag3WMcQDbfdnFiZBi2Xc3zbwmgTT1bL9UNtWQ@mail.gmail.com>
 <CAOcnVr1Df1TXCFpvKTip1jrTc3fhSOuRCOD7Evip7Dd=Shp_vQ@mail.gmail.com>
 <CAO7hTbMR7LNpxLMOTkz39WSUFcpNTc6-3OMbMA32Y5n3mcgf2g@mail.gmail.com>
 <CAO7hTbMNT4_wJHOn3cSKaQ7Bt9LOgFV+HdfsnYubU5VBqt-nYg@mail.gmail.com>
 <CAO7hTbNTfGtPNE1EkbaGcERuhnuW+5Cgush+AN6HNnO6+sVrYQ@mail.gmail.com>
 <1367850274.24846.YahooMailNeo@web125601.mail.ne1.yahoo.com>
From: Harsh J <harsh@cloudera.com>
Date: Tue, 7 May 2013 13:14:30 +0530
Message-ID: 
 <CAOcnVr07pGivfqTRTqq4Rv-wDX11REcj9M+hnAs47jwEQVh6GA@mail.gmail.com>
Subject: Re: no _SUCCESS file in MR output directory.
To: "user@oozie.apache.org" <user@oozie.apache.org>,
 Eduardo Afonso Ferreira <eafonsof@yahoo.com>
Cc: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Good observance: Pig does seem to use a default "false" when possible,
to disable the _SUCCESS creation. I don't see Hive do that, nor any
part of the stock Apache Hadoop MR jobs.

Rahul - Do you use a Pig action in your WF? Also, are you definitively
seeing _SUCCESS being created after you add the option manually?

On Mon, May 6, 2013 at 7:54 PM, Eduardo Afonso Ferreira
<eafonsof@yahoo.com> wrote:
> I'm not sure if Pig disables it or not, but I remember I had issues when =
that file was to be created by the MR jobs due to the fact of Oozie or Pig =
removing temporary directories or something like that. I remember seeing an=
 exception about failure to create the SUCCESS file, so I started using the=
 following property in my workflow pig action to disable that:
>
>         <pig>
>             ...
>             <configuration>
>
>                 <property>
>                     <name>mapreduce.fileoutputcommitter.marksuccessfuljob=
s</name>
>                     <value>false</value>
>                 </property>
>                 ...
>             </configuration>
>             ...
>         </pig>
>
>
>
> ________________________________
>  From: Rahul Bhattacharjee <rahul.rec.dgp@gmail.com>
> To: "user@hadoop.apache.org" <user@hadoop.apache.org>; user@oozie.apache.=
org
> Sent: Monday, May 6, 2013 3:48 AM
> Subject: Re: no _SUCCESS file in MR output directory.
>
>
> I wanted to confirm whether oozie disables the _SUCCESS file creation whe=
n
> it triggers a MR job.
>
> I am triggering a MR job (actually a bunch of 'em) from oozie and the
> workflow completes successfully , however I do not see any kind of _SUCCE=
SS
> file in the output directory.
> When I set the file output committer's configuration
> (mapreduce.fileoutputcommitter.
> marksuccessfuljobs
> ) to true.It generates the success file. Wanted to confirm if oozie does
> the disabling of success file creation.
>
> Thanks,
> Rahul
>
>
> On Mon, May 6, 2013 at 12:34 PM, Rahul Bhattacharjee <
> rahul.rec.dgp@gmail.com> wrote:
>
>> Oozie is being used for triggering the MR job. Looks like oozie disables
>> the success file creation using the configuration that you have mentione=
d
>> for FileOutputCommitter.
>>
>> I have enabled it by setting this property in conf.
>>
>> Rahul
>>
>>
>> On Mon, May 6, 2013 at 9:38 AM, Rahul Bhattacharjee <
>> rahul.rec.dgp@gmail.com> wrote:
>>
>>> Thanks Harsh for the pointers. I will find out more on this.
>>>
>>>
>>> On Sun, May 5, 2013 at 11:26 PM, Harsh J <harsh@cloudera.com> wrote:
>>>
>>>> I can think of a few, most obvious ones:
>>>>
>>>> 1. Job didn't succeed and/or the file was deleted (*shields self*)
>>>> 2. Job overrode the default FileOutputCommitter with something that
>>>> doesn't do success marking.
>>>> 3. Job specifically asked to not create such files, via config
>>>> mapreduce.fileoutputcommitter.marksuccessfuljobs or so, set to false.
>>>>
>>>> On Sun, May 5, 2013 at 9:54 PM, Rahul Bhattacharjee
>>>> <rahul.rec.dgp@gmail.com> wrote:
>>>> > Hi,
>>>> >
>>>> >
>>>> > A few days back , I was going through a MR job's output , but there
>>>> wasn't
>>>> > any _SUCCESS file in the output directory.
>>>> > I was wondering what all reasons for this  (no _SUCCESS file)?
>>>> >
>>>> > Thanks,
>>>> > Rahul
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>


--=20
Harsh J