hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Gates <ga...@yahoo-inc.com>
Subject Re: the last job in the mapreduce plan
Date Tue, 15 Jun 2010 16:26:15 GMT
I've never seen a case where this happens.  Is this a theoretical  
question or are you seeing this issue?


On Jun 15, 2010, at 8:49 AM, Gang Luo wrote:

> Hi,
> Is it possible the last MapReduce job in the MR plan only loads  
> something and stores it without any other processing in between? For  
> example, when visiting some physical operator, we need to end the  
> current MR operator after embedding the physical operator into MR  
> operator, and create a new MR operator for later physical operators.  
> Unfortunately, the following physical operator is a store, the end  
> of the entire query. In this case, the last MR operator only contain  
> load and store without any meaningful work in between. This idle  
> MapReduce job will degrade the performance. Will this happen in Pig?
> Thanks,
> -Gang

View raw message