apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Weise (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-599) Data not fully processed when operator terminates via ShutdownException
Date Wed, 04 Jan 2017 18:31:58 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798979#comment-15798979

Thomas Weise commented on APEXCORE-599:

I could reproduce it on a Ubuntu VM by picking a specific test and run it long enough like

while mvn test -Dtest=ParDoBoundTranslatorTest#testMultiOutputParDoWithSideInputs; do :; done

Enable tuple tracing in the pipeline options and it will make it easier to see how the last
tuple even though it was emitted by the input operator in the streaming window *before* shutdown
never makes it downstream.

> Data not fully processed when operator terminates via ShutdownException
> -----------------------------------------------------------------------
>                 Key: APEXCORE-599
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-599
>             Project: Apache Apex Core
>          Issue Type: Bug
>    Affects Versions: 3.5.0
>            Reporter: Thomas Weise
> Observed in the form of flaky unit tests in Beam. The pipelines terminate using shutdown
by the operator when there is no more input and occasionally the test will fail because the
cluster exits without having processed fully. The bandaid is to have a delay before shutdown
is issued, allowing the last tuples to be pushed downstream. See BEAM-1140 for details.

This message was sent by Atlassian JIRA

View raw message