hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Chauhan <ashutosh.chau...@gmail.com>
Subject Re: the last job in the mapreduce plan
Date Wed, 16 Jun 2010 16:30:10 GMT
Without knowing much about your operator, I will suggest couple of
things. In general, you should avoid designing operators which
requires you to take explicit actions on pipeline (which in your case
requires pipeline to be closed immediately). Currently there are
operators in Pig which actually does that but we should carefully
consider adding more as there interactions in pipeline leads to
special cases.
If thats not feasible, then one option for you could be to write a
visitor which will traverse the generated MR plan. If it finds your
operator in the pipeline, it will look if there is any more MR
operator following the current one and if its safe to remove that or
readjust the pipeline.
This may be more complicated then what it needs to be, if you can shed
more light on your operator I may be able to suggest a better

Hope it helps,

On Wed, Jun 16, 2010 at 07:20, Gang Luo <lgpublic@yahoo.com.cn> wrote:
> Thanks for replying. Actually, I didn't observe such thing happen in pig now. But one
of the operators I implement in Pig requires to end the current MR operator afterwards. That
issue may happen in my case.
> -Gang
> ----- 原始邮件 ----
> 发件人: Ashutosh Chauhan <ashutosh.chauhan@gmail.com>
> 收件人: pig-dev@hadoop.apache.org
> 发送日期: 2010/6/15 (周二) 1:24:46 下午
> 主   题: Re: the last job in the mapreduce plan
> Gang,
> What you are saying can never happen because we create a new MR
> operator only when we have a blocking operator which needs to go in
> the next MR operator. We dont create new MR operator apriori without
> looking at next physical operator in the pipeline. If you are seeing
> this happening, I would consider that as a bug.
> Ashutosh
> On Tue, Jun 15, 2010 at 09:26, Alan Gates <gates@yahoo-inc.com> wrote:
>> I've never seen a case where this happens. 營s this a theoretical question
>> or are you seeing this issue?
>> Alan.
>> On Jun 15, 2010, at 8:49 AM, Gang Luo wrote:
>>> Hi,
>>> Is it possible the last MapReduce job in the MR plan only loads something
>>> and stores it without any other processing in between? For example, when
>>> visiting some physical operator, we need to end the current MR operator
>>> after embedding the physical operator into MR operator, and create a new MR
>>> operator for later physical operators. Unfortunately, the following physical
>>> operator is a store, the end of the entire query. In this case, the last MR
>>> operator only contain load and store without any meaningful work in between.
>>> This idle MapReduce job will degrade the performance. Will this happen in
>>> Pig?
>>> Thanks,
>>> -Gang

View raw message