Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of haavard.kongsgaard@gmail.com
 designates 209.85.216.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <022f01cdeb4e$ea7fc170$bf7f4450$@yahoo.com>
References: 
 <CAKH910_r5y080-bMj6xQ8d9o5tFfkitjQ9kuQgYNGA4m-YP=ig@mail.gmail.com>
	<022f01cdeb4e$ea7fc170$bf7f4450$@yahoo.com>
Date: Sat, 5 Jan 2013 17:48:58 +0100
Message-ID: 
 <CAKH910_27yiqBKVb6MAHWD1AjtRwqYTFrWo=rV-j3E+4imxC=w@mail.gmail.com>
Subject: Re: Skipping entire task
From: =?ISO-8859-1?Q?H=E5vard_Wahl_Kongsg=E5rd?=
 <haavard.kongsgaard@gmail.com>
To: user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

yes, but I use pydoop not the native java library. The problem is that
the same task times, so a solution is not that straightforward. And
Pydoop does not seem to have any methods to inform the task how many
times it has failed. So if there is no native method in hadoop, I
could use a database or something for that purpose. Any other ideas?

-H=E5vard

On Sat, Jan 5, 2013 at 3:14 PM, David Parks <davidparks21@yahoo.com> wrote:
> Thinking here... if you submitted the task programmatically you should be
> able to capture the failure of the task and gracefully move past it to yo=
ur
> next tasks.
>
> To say it in a long-winded way:  Let's say you submit a job to Hadoop, a
> java jar, and your main class implements Tool. That code has the
> responsibility to submit a series of jobs to hadoop, something like this:
>
> try{
>   Job myJob =3D new MyJob(getConf());
>   myJob.submitAndWait();
> }catch(Exception uhhohh){
>   //Deal with the issue and move on
> }
> Job myNextJob =3D new MyNextJob(getConf());
> myNextJob.submit();
>
> Just pseudo code there to demonstrate my thought.
>
> David
>
>
>
> -----Original Message-----
> From: H=E5vard Wahl Kongsg=E5rd [mailto:haavard.kongsgaard@gmail.com]
> Sent: Saturday, January 05, 2013 4:54 PM
> To: user
> Subject: Skipping entire task
>
> Hi, hadoop can skip bad records
> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittl=
e-c
> ode.
> But it is also possible to skip entire tasks?
>
> -H=E5vard
>
> --
> H=E5vard Wahl Kongsg=E5rd
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
>


--=20
H=E5vard Wahl Kongsg=E5rd
Faculty of Medicine &
Department of Mathematical Sciences
NTNU

http://havard.security-review.net/