Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.210.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKH9109QZqV7kARAaGDJ-nmZRKJxE3B0VCJjRzOO19N2=pZnow@mail.gmail.com>
References: 
 <CAKH910_r5y080-bMj6xQ8d9o5tFfkitjQ9kuQgYNGA4m-YP=ig@mail.gmail.com>
 <022f01cdeb4e$ea7fc170$bf7f4450$@yahoo.com>
 <CAEAKFL9qQ=00F0TbchJkYbzE+DDw6Yw7JwSQ7m0fc8oJSKAr1A@mail.gmail.com>
 <CAKH9109QZqV7kARAaGDJ-nmZRKJxE3B0VCJjRzOO19N2=pZnow@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Sun, 6 Jan 2013 20:16:58 +0530
Message-ID: 
 <CAOcnVr1_EvvmPjtPZuvNvkkJK3pG8BL+CqEVPROJ5CvujV14EQ@mail.gmail.com>
Subject: Re: Skipping entire task
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You can use the mapred.max.map.failures.percent and
mapred.max.reduce.failures.percent features to control the percentage
of allowed failures of tasks in a single job (despite which the job is
marked successful).

On Sun, Jan 6, 2013 at 8:04 PM, H=E5vard Wahl Kongsg=E5rd
<haavard.kongsgaard@gmail.com> wrote:
>> Are tasks being executed multiple times due to failures? Sorry, it was n=
ot
>> very clear from your question.
>
> yes, and I simply want to skip them if they fail more than x
> times(after all this is big data :) ).
>
> -H=E5vard
>
> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
> <yhemanth@thoughtworks.com> wrote:
>> Hi,
>>
>> Are tasks being executed multiple times due to failures? Sorry, it was n=
ot
>> very clear from your question.
>>
>> Thanks
>> hemanth
>>
>>
>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <davidparks21@yahoo.com> wro=
te:
>>>
>>> Thinking here... if you submitted the task programmatically you should =
be
>>> able to capture the failure of the task and gracefully move past it to
>>> your
>>> next tasks.
>>>
>>> To say it in a long-winded way:  Let's say you submit a job to Hadoop, =
a
>>> java jar, and your main class implements Tool. That code has the
>>> responsibility to submit a series of jobs to hadoop, something like thi=
s:
>>>
>>> try{
>>>   Job myJob =3D new MyJob(getConf());
>>>   myJob.submitAndWait();
>>> }catch(Exception uhhohh){
>>>   //Deal with the issue and move on
>>> }
>>> Job myNextJob =3D new MyNextJob(getConf());
>>> myNextJob.submit();
>>>
>>> Just pseudo code there to demonstrate my thought.
>>>
>>> David
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: H=E5vard Wahl Kongsg=E5rd [mailto:haavard.kongsgaard@gmail.com]
>>> Sent: Saturday, January 05, 2013 4:54 PM
>>> To: user
>>> Subject: Skipping entire task
>>>
>>> Hi, hadoop can skip bad records
>>>
>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brit=
tle-c
>>> ode.
>>> But it is also possible to skip entire tasks?
>>>
>>> -H=E5vard
>>>
>>> --
>>> H=E5vard Wahl Kongsg=E5rd
>>> Faculty of Medicine &
>>> Department of Mathematical Sciences
>>> NTNU
>>>
>>> http://havard.security-review.net/
>>>
>>
>
>
>
> --
> H=E5vard Wahl Kongsg=E5rd
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/


--=20
Harsh J