crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul <>
Subject Re: Pipeline in synchronous mode
Date Wed, 27 Jun 2012 11:10:29 GMT
Hi Gabried,

Thanks this helps. Actually I was getting some exceptions but was unable 
to debug them as the tests got finished outputting the stacktraces.
Later realized some thing silly on my end.


On 27-06-2012 15:10, Gabriel Reid wrote:
> On Wed, Jun 27, 2012 at 10:12 AM, Rahul<>  wrote:
>> How can I run Pipeline in synchronous mode ?  I have built junit tests but
>> they all run in async mode over MRPipeline.
>> I need to make sure that the pipeline gets finished so I have to continually
>> monitor the output folder.  Is there a better way for this ?
> Hi Rahul,
> Pipelines themselves are always run synchronously; that is, a call to
> or Pipeline.done will block until all underlying jobs have
> run (or failed).
> There shouldn't be a need to monitor output folders; you can count
> on output directories being populated once or
> Pipeline.done returns (and shouldn't count on the folders being populated
> until that point.
> Of course, the underlying MapReduce job(s) are run asynchronously.
> The coordination of the asynchronous MR jobs and the blocking
> of the MRPipeline class is handled by the CrunchJobControl class.
> Hope this helps.
> Regards,
> Gabriel

View raw message