hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Ulrik S√łttrup <soett...@nbi.dk>
Subject Re: iterative map-reduce
Date Tue, 29 Jul 2008 15:03:56 GMT
Hi Shirley,

I am basically doing as Qin suggested.
I am running a job iteratively until some condition is met.
My main looks something like:(in pseudo code)

main:
 while (!converged):
   make new jobconf
   setup jobconf
   run jobconf
   check reporter for statistics
   decide if converged

I use a custom reporter to check on the fitness of the solution in the 
reduce phase.

If you need more(real java) code drop me a line.

Cheers,
Christian

Qin Gao wrote:
> I think it is nothing to do with the framework, just treat the mapredcue as
> a batch process or a subroutine, and you may iteratively call them. If there
> are such interface, I am also interested to know.
>
>
>
> On Tue, Jul 29, 2008 at 10:31 AM, Shirley Cohen <shirleyc@cis.upenn.edu>wrote:
>
>   
>> Thanks... would the iterative script be run outside of Hadoop? I was
>> actually trying to figure out if the framework could handle iterations.
>>
>> Shirley
>>
>>
>> On Jul 29, 2008, at 9:10 AM, Qin Gao wrote:
>>
>>  if you are using java, just create job configure again and run it,
>>     
>>> otherwise
>>> you just need to write a iterative script.
>>>
>>> On Tue, Jul 29, 2008 at 9:57 AM, Shirley Cohen <scohen@cs.utexas.edu>
>>> wrote:
>>>
>>>  Hi,
>>>       
>>>> I want to call a map-reduce program recursively until some condition is
>>>> met.  How do I do that?
>>>>
>>>> Thanks,
>>>>
>>>> Shirley
>>>>
>>>>
>>>>
>>>>         
>
>   


Mime
View raw message