hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gang Luo <lgpub...@yahoo.com.cn>
Subject the last job in the mapreduce plan
Date Tue, 15 Jun 2010 15:49:15 GMT
Is it possible the last MapReduce job in the MR plan only loads something and stores it without
any other processing in between? For example, when visiting some physical operator, we need
to end the current MR operator after embedding the physical operator into MR operator, and
create a new MR operator for later physical operators. Unfortunately, the following physical
operator is a store, the end of the entire query. In this case, the last MR operator only
contain load and store without any meaningful work in between. This idle MapReduce job will
degrade the performance. Will this happen in Pig?



View raw message