Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 875E010B29 for ; Mon, 6 May 2013 19:13:04 +0000 (UTC) Received: (qmail 10573 invoked by uid 500); 6 May 2013 19:12:59 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 10419 invoked by uid 500); 6 May 2013 19:12:58 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Delivered-To: moderator for user@hadoop.apache.org Received: (qmail 42082 invoked by uid 99); 6 May 2013 14:25:03 -0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 85623.48222.bm@omp1063.mail.ne1.yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1367850274; bh=kY86uzmXjuBssWeAv45AnMJoqj5CE1JzP0DYQBt5FR4=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=hAhpA/6j8q1NBrMH4MZTxNoCVDwCQ3v6b3MekgAh989RgHjO1L/xQ1vMyMmaYLXTXu6Rcx6NFXpAvQ0ILEzom+FOwcjHXKS5+rK0C1AR6JeTALPnkyS5uW7h5eCEgJ2BmCgGD7rAYnLdR4uqL1Yvxl/yLaawHtDvDxNFubpyzzA= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=6IY7AJNyq1ZeMBGJHxVlUuEb4u4KwJojTGEoKwyVr2L9PXzVCERE2u9GSU1zH64FAooYDjT0YVpSkcw/X3vOqo0/DASqFVA0ybiC/+vzqrayHhMxLawrIGClOI8QstyDddbnCTmTorQkJnO9SqIDa5NmFFg1c/r9FP5vvHm0Fvs=; X-YMail-OSG: RvC_ZEMVM1mKyNmuB.xwbVqJ6r.3XI0D7mJgYcn2B7CkzOi jIHQnwEGLkp6ycy8LB7DuTXgv1ovqE0FScPw7oXOoJet_hT3Q84Xs1J0I2zL hf1R2_y1qnkOzTq.KhsT5nBAocBtgeljJ3N.sdcgX89ohzBxjlM7val3.Jbh gtKykbwBbLy5jSMR6bkMZEmlViuCJv2TfwDOp.pC0x3Yd4QSOi.oh3lrP02W gupcBbFgsr3fz5WPVU4zUPJAY27mIprTptDS27X9JWLDhDFE8QIcidwpevf2 5Hnttk60A2ujbO0GViIpoKGgQLHYmmgziHDMCojY7gMr_3M_ZW4sl9cXXDNl cYQiBDokRJZLyeo__csP7zeX3khD2FdwiLT6aEI51YplvSuOvkHCbUICqMeQ TkDg7pXaAMoXd3Ucv.ZnmsUFEWZ1EXQHp6RE29RGZQSfL1f1ORNtE0Tfhd3K nMk8le6R5E2.i9KTrD_d9wBHNzFshRF1J2RMIJpjpHdne_c_cgJqxYjWs9RG LHjU.r60f3w84N9zlVmGsCYowmM_uZUwYP6KFi968BlWPjqDaLvNsIVKtG_K hiIoAJRs- X-Rocket-MIMEInfo: 002.001,SSdtIG5vdCBzdXJlIGlmIFBpZyBkaXNhYmxlcyBpdCBvciBub3QsIGJ1dCBJIHJlbWVtYmVyIEkgaGFkIGlzc3VlcyB3aGVuIHRoYXQgZmlsZSB3YXMgdG8gYmUgY3JlYXRlZCBieSB0aGUgTVIgam9icyBkdWUgdG8gdGhlIGZhY3Qgb2YgT296aWUgb3IgUGlnIHJlbW92aW5nIHRlbXBvcmFyeSBkaXJlY3RvcmllcyBvciBzb21ldGhpbmcgbGlrZSB0aGF0LiBJIHJlbWVtYmVyIHNlZWluZyBhbiBleGNlcHRpb24gYWJvdXQgZmFpbHVyZSB0byBjcmVhdGUgdGhlIFNVQ0NFU1MgZmlsZSwgc28gSSBzdGFydGVkIHUBMAEBAQE- X-Mailer: YahooMailWebService/0.8.141.536 References: Message-ID: <1367850274.24846.YahooMailNeo@web125601.mail.ne1.yahoo.com> Date: Mon, 6 May 2013 07:24:34 -0700 (PDT) From: Eduardo Afonso Ferreira Reply-To: Eduardo Afonso Ferreira Subject: Re: no _SUCCESS file in MR output directory. To: "user@oozie.apache.org" , "user@hadoop.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-1981468715-83905784-1367850274=:24846" X-Virus-Checked: Checked by ClamAV on apache.org ---1981468715-83905784-1367850274=:24846 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I'm not sure if Pig disables it or not, but I remember I had issues when th= at file was to be created by the MR jobs due to the fact of Oozie or Pig re= moving temporary directories or something like that. I remember seeing an e= xception about failure to create the SUCCESS file, so I started using the f= ollowing property in my workflow pig action to disable that:=0A=0A=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ...= =0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =0A=0A=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =0A=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 mapreduce.fi= leoutputcommitter.marksuccessfuljobs=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 false=0A=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =0A=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ...=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= ...=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 =0A=0A=0A=0A______________________= __________=0A From: Rahul Bhattacharjee =0ATo: "us= er@hadoop.apache.org" ; user@oozie.apache.org =0ASe= nt: Monday, May 6, 2013 3:48 AM=0ASubject: Re: no _SUCCESS file in MR outpu= t directory.=0A =0A=0AI wanted to confirm whether oozie disables the _SUCCE= SS file creation when=0Ait triggers a MR job.=0A=0AI am triggering a MR job= (actually a bunch of 'em) from oozie and the=0Aworkflow completes successf= ully , however I do not see any kind of _SUCCESS=0Afile in the output direc= tory.=0AWhen I set the file output committer's configuration=0A(mapreduce.f= ileoutputcommitter.=0Amarksuccessfuljobs=0A) to true.It generates the succe= ss file. Wanted to confirm if oozie does=0Athe disabling of success file cr= eation.=0A=0A=E2=80=8BThanks,=0ARahul=E2=80=8B=0A=0A=0AOn Mon, May 6, 2013 = at 12:34 PM, Rahul Bhattacharjee <=0Arahul.rec.dgp@gmail.com> wrote:=0A=0A>= Oozie is being used for triggering the MR job. Looks like oozie disables= =0A> the success file creation using the configuration that you have mentio= ned=0A> for FileOutputCommitter.=0A>=0A> I have enabled it by setting this = property in conf.=0A>=0A> Rahul=0A>=0A>=0A> On Mon, May 6, 2013 at 9:38 AM,= Rahul Bhattacharjee <=0A> rahul.rec.dgp@gmail.com> wrote:=0A>=0A>> Thanks = Harsh for the pointers. I will find out more on this.=0A>>=0A>>=0A>> On Sun= , May 5, 2013 at 11:26 PM, Harsh J wrote:=0A>>=0A>>> I= can think of a few, most obvious ones:=0A>>>=0A>>> 1. Job didn't succeed a= nd/or the file was deleted (*shields self*)=0A>>> 2. Job overrode the defau= lt FileOutputCommitter with something that=0A>>> doesn't do success marking= .=0A>>> 3. Job specifically asked to not create such files, via config=0A>>= > mapreduce.fileoutputcommitter.marksuccessfuljobs or so, set to false.=0A>= >>=0A>>> On Sun, May 5, 2013 at 9:54 PM, Rahul Bhattacharjee=0A>>> wrote:=0A>>> > Hi,=0A>>> >=0A>>> >=0A>>> > A few days bac= k , I was going through a MR job's output , but there=0A>>> wasn't=0A>>> > = any _SUCCESS file in the output directory.=0A>>> > I was wondering what all= reasons for this=C2=A0 (no _SUCCESS file)?=0A>>> >=0A>>> > Thanks,=0A>>> >= Rahul=0A>>>=0A>>>=0A>>>=0A>>> --=0A>>> Harsh J=0A>>>=0A>>=0A>>=0A> ---1981468715-83905784-1367850274=:24846 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
I'm not sure if Pig disables it or not, but I remembe= r I had issues when that file was to be created by the MR jobs due to the f= act of Oozie or Pig removing temporary directories or something like that. = I remember seeing an exception about failure to create the SUCCESS file, so= I started using the following property in my workflow pig action to disabl= e that:

        <pig>
  &nbs= p;         ...
            <configuration>
         = ;       <property>
<= font size=3D"2">                  =   <name>mapreduce.fileoutputcommitter.marksuccessfuljobs</nam= e>
    &nbs= p;               <value>false</= value>
    =             </property>
          &nbs= p;     ...
        &nb= sp;   </configuration>
          &nb= sp; ...
        </pi= g>


= From= : Rahul Bhattacharjee <rahul.rec.dgp@gmail.com>
To: "user@hadoop.apache.org" <= ;user@hadoop.apache.org>; user@oozie.apache.org
Sent: Monday, May 6, 2013 3:48 AM
Subject: Re: no _SUCCESS file in = MR output directory.

= I wanted to confirm whether oozie disables the _SUCCESS file creation when<= br>it triggers a MR job.

I am triggering a MR job (actually a bunch = of 'em) from oozie and the
workflow completes successfully , however I d= o not see any kind of _SUCCESS
file in the output directory.
When I s= et the file output committer's configuration
(mapreduce.fileoutputcommitter.
marksuccessfuljobs
= ) to true.It generates the success file. Wanted to confirm if oozie doesthe disabling of success file creation.

=E2=80=8BThanks,
Rahul= =E2=80=8B


On Mon, May 6, 2013 at 12:34 PM, Rahul Bhattacharjee &= lt;
rahul.rec.dgp@gmail.com> wrote:

> Oozie = is being used for triggering the MR job. Looks like oozie disables
> = the success file creation using the configuration that you have mentioned> for FileOutputCommitter.
>
> I have enabled it by settin= g this property in conf.
>
> Rahul
>
>
> On M= on, May 6, 2013 at 9:38 AM, Rahul Bhattacharjee <
> rah= ul.rec.dgp@gmail.com> wrote:
>
>> Thanks Harsh for th= e pointers. I will find out more on this.
>>
>>
>> On Sun, Ma= y 5, 2013 at 11:26 PM, Harsh J <harsh@cloudera.com> wrote:
&g= t;>
>>> I can think of a few, most obvious ones:
>>= >
>>> 1. Job didn't succeed and/or the file was deleted (*sh= ields self*)
>>> 2. Job overrode the default FileOutputCommitte= r with something that
>>> doesn't do success marking.
>&g= t;> 3. Job specifically asked to not create such files, via config
&g= t;>> mapreduce.fileoutputcommitter.marksuccessfuljobs or so, set to f= alse.
>>>
>>> On Sun, May 5, 2013 at 9:54 PM, Rahul= Bhattacharjee
>>> <rahul.rec.dgp@gmail.com&g= t; wrote:
>>> > Hi,
>>> >
>>> >
>>> > A few days back , I was going through a MR job's= output , but there
>>> wasn't
>>> > any _SUCCES= S file in the output directory.
>>> > I was wondering what a= ll reasons for this  (no _SUCCESS file)?
>>> >
>&= gt;> > Thanks,
>>> > Rahul
>>>
>>= >
>>>
>>> --
>>> Harsh J
>>= >
>>
>>
>

---1981468715-83905784-1367850274=:24846--