beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aviem Zur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner
Date Tue, 08 Aug 2017 15:17:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118451#comment-16118451
] 

Aviem Zur commented on BEAM-2671:
---------------------------------

[~jkff] we've been running Spark runner in streaming mode using 2.0.0 
We've encountered the issue with the watermarks not advancing which was a no-go for us, since
results were not emitted and eventually apps failed on out of memory exceptions. This was
the reason for the fix in https://github.com/apache/beam/pull/3343

After integrating this fix (on our local fork) we were able to run applications successfully.
Regarding running the current state of master, we are not, so hard to comment on if the failing
tests cause real applications to fail, this will have to be assessed. What we do know is the
tests are failing.

Regarding ValidatesRunner tests, while these do pass, IIRC the creation of the streaming tests
for Spark runner were in fact due to insufficient tests for Spark runner's streaming mode.
So while they are not perfect and flaky, they do provide value. Perhaps [~amitsela] can elaborate
on these tests.

> CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner
> --------------------------------------------------------------------------------
>
>                 Key: BEAM-2671
>                 URL: https://issues.apache.org/jira/browse/BEAM-2671
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Etienne Chauchot
>            Assignee: Jean-Baptiste Onofré
>             Fix For: 2.2.0
>
>
> Error message:
> Flatten.Iterables/FlattenIterables/FlatMap/ParMultiDo(Anonymous).out0: 
> Expected: iterable over [] in any order
>      but: Not matched: "late"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message