pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xianda Ke (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4876) OutputConsumeIterator can't handle the last buffered tuples for some Operators
Date Mon, 25 Apr 2016 03:07:12 GMT

    [ https://issues.apache.org/jira/browse/PIG-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255821#comment-15255821
] 

Xianda Ke commented on PIG-4876:
--------------------------------

I think your proposal is similar with the solution in PIG-4842 ([PIG-4842-2.patch|https://issues.apache.org/jira/secure/attachment/12795373/PIG-4842-2.patch])
and PIG-4857( [PIG-4857.patch|https://issues.apache.org/jira/secure/attachment/12795997/PIG-4857.patch]),
which just reset the flag and does not touch non-spark code.
it should work if we refactor the code by implementing the abstract method(like beginOfInput()).
 

Yes, it minimize non-spark code change, but it seem a bit tricky. 



> OutputConsumeIterator can't handle the last buffered tuples for some Operators
> ------------------------------------------------------------------------------
>
>                 Key: PIG-4876
>                 URL: https://issues.apache.org/jira/browse/PIG-4876
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: Xianda Ke
>            Assignee: Xianda Ke
>             Fix For: spark-branch
>
>         Attachments: PIG-4876.patch
>
>
> Some Operators, such as MergeCogroup, Stream, CollectedGroup etc buffer some input records
to constitute the result tuples. The last result tuples are buffered in the operator.  These
Operators need a flag to indicate the end of input, so that they can flush and constitute
their last tuples.
> Currently, the flag 'parentPlan.endOfAllInput' is targeted for flushing the buffered
tuples in MR mode.  But it does not work with OutputConsumeIterator in Spark mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message