hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim the Standing Bear" <standingb...@gmail.com>
Subject Re: can jobs be launched recursively within a mapper ?
Date Mon, 29 Oct 2007 21:17:25 GMT
Hi Lohit,

Thank you for the response.  If jobs cannot be recursively launched,
does it mean that I have to pre-process all the catalogs and putting
only filenames into a sequence file and then use hadoop from that
point on?

-- Jim

On 10/29/07, lohit.vijayarenu@yahoo.com <lohit.vijayarenu@yahoo.com> wrote:
> >Can I recursively launch a job within the map() method of a mapper?
>
> Interesting. If your launch is blocking, then I am thinking possibility of deadlock.
Lets say you have N nodes and your initial map tasks are scheduled on all N nodes. Now if
each of the map tasks launch few more jobs, then the new ones are sitting there waiting for
free nodes.
>
> Thanks,
> Lohit
> ----- Original Message ----
> From: Jim the Standing Bear <standingbear@gmail.com>
> To: hadoop-user@lucene.apache.org
> Sent: Monday, October 29, 2007 1:28:17 PM
> Subject: can jobs be launched recursively within a mapper ?
>
>
> Hi,
>
> Can I recursively launch a job within the map() method of a mapper?
> The task i am facing involves extracting metadata for filenames and
> catalog names from hierarchical catalogs, and for each catalog name
> extract, I must recursively do the same thing until no more catalog
> names present.  Therefore, it will be convenient if I can launch a job
> within the map(), but I want to make certain that it will work before
> investing too much efforts into the coding.
>
> If it won't work, could you suggest the proper way to go about doing
> so?  Much appreciated in advance.
>
> -- Jim
>
>
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>


-- 
--------------------------------------
Standing Bear Has Spoken
--------------------------------------

Mime
View raw message