hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Dai <jiany...@yahoo-inc.com>
Subject Re: explicitly close a mr job
Date Tue, 20 Jul 2010 19:38:03 GMT
You can refer to MrCompiler.startNew. You need to add store to close 
current MapReduceOper, create a new MapReduceOper, add load, then add 
MapReduceOper to MRPlan.


Gang Luo wrote:
> Hi all,
> when compile a physical plan into MR plan, the current rule is to put as many 
> operator as possible into the reduce phase of the current mr job. But sometimes 
> we want to control over this in physical plan. Say we want to put operator 1 
> into reduce phase of current mr job, end it and then put operator 2 into map 
> phase of the next mr job (both operator 1 & 2 are non-blocking). It seems 
> inserting store and load operator in physical plan doesn't help. Is there a 
> better way to do this than implementing new operators )e.g. starter and ender)
> Thanks,
> -Gang

View raw message