Hi Lohit,
Thank you for the response. If jobs cannot be recursively launched,
does it mean that I have to pre-process all the catalogs and putting
only filenames into a sequence file and then use hadoop from that
point on?
-- Jim
On 10/29/07, lohit.vijayarenu@yahoo.com <lohit.vijayarenu@yahoo.com> wrote:
> >Can I recursively launch a job within the map() method of a mapper?
>
> Interesting. If your launch is blocking, then I am thinking possibility of deadlock.
Lets say you have N nodes and your initial map tasks are scheduled on all N nodes. Now if
each of the map tasks launch few more jobs, then the new ones are sitting there waiting for
free nodes.
>
> Thanks,
> Lohit
> ----- Original Message ----
> From: Jim the Standing Bear <standingbear@gmail.com>
> To: hadoop-user@lucene.apache.org
> Sent: Monday, October 29, 2007 1:28:17 PM
> Subject: can jobs be launched recursively within a mapper ?
>
>
> Hi,
>
> Can I recursively launch a job within the map() method of a mapper?
> The task i am facing involves extracting metadata for filenames and
> catalog names from hierarchical catalogs, and for each catalog name
> extract, I must recursively do the same thing until no more catalog
> names present. Therefore, it will be convenient if I can launch a job
> within the map(), but I want to make certain that it will work before
> investing too much efforts into the coding.
>
> If it won't work, could you suggest the proper way to go about doing
> so? Much appreciated in advance.
>
> -- Jim
>
>
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
--
--------------------------------------
Standing Bear Has Spoken
--------------------------------------
|