hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim the Standing Bear" <standingb...@gmail.com>
Subject Re: can jobs be launched recursively within a mapper ?
Date Tue, 30 Oct 2007 01:57:22 GMT
Thanks, Stu...  Maybe my mind is way off track - but I still sense a
problem with the mapper sending feedbacks to the job controller.  That
is, when a mapper has reached the terminal condition, how can it tell
the job controller to stop?

If I keep a JobConf object in the mapper, and set a property
"stop.processing" to true when a mapping task has reached the terminal
condition, will it cause synchronization problems?  There could be
other mapping tasks that still wish to go on?

I tried to find a way so that the job controller can open the file in
the output path at the end of the loop to read the contents; but thus
far, I haven't seen a way to achieve this.

Does this mean I have hit a dead-end?

-- Jim



On 10/29/07, Stu Hood <stuhood@webmail.us> wrote:
> The iteration would take place in your control code (your 'main' method, as shown in
the examples).
>
> In order to prevent records from looping infinitely, each iteration would need to use
a separate output/input directory.
>
> Thanks,
> Stu
>
>
> -----Original Message-----
> From: Jim the Standing Bear <standingbear@gmail.com>
> Sent: Monday, October 29, 2007 5:45pm
> To: hadoop-user@lucene.apache.org
> Subject: Re: can jobs be launched recursively within a mapper ?
>
> thanks, Owen and David,
>
> I also thought of making a queue so that I can push catalog names to
> the end of it, while the job control loop keeps removing items off the
> queue until there is no more left.
>
> However, the problem is I don't see how I can do so within the
> map/reduce context.  All the code examples are one-shot deals and
> there is no iteration involved.
>
> Furthermore, what David said made sense, but to avoid infinite loop,
> the code must remove the record it just read from the input file.  How
> do I do that using hadoop's fs?  or does hadoop take care of it
> automatically?
>
> -- Jim
>
>
>
> On 10/29/07, David Balatero <ezwelty@u.washington.edu> wrote:
> > Aren't these questions a little advanced for a bear to be asking?
> > I'll be here all night...
> >
> > But seriously, if your job is inherently recursive, one possible way
> > to do it would be to make sure that you output in the same format
> > that you input. Then you can keep re-reading the outputted file back
> > into a new map/reduce job, until you hit some base case and you
> > terminate. I've had a main method before that would kick off a bunch
> > of jobs in a row -- but I wouldn't really recommend starting another
> > map/reduce job in the scope of a running map() or reduce() method.
> >
> > - David
> >
> >
> > On Oct 29, 2007, at 2:17 PM, Jim the Standing Bear wrote:
> >
> > > then
> >
> >
>
>
> --
> --------------------------------------
> Standing Bear Has Spoken
> --------------------------------------
>
>
>


-- 
--------------------------------------
Standing Bear Has Spoken
--------------------------------------

Mime
View raw message