beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-1130) SparkRunner ResumeFromCheckpointStreamingTest Failing
Date Mon, 12 Dec 2016 10:48:58 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741603#comment-15741603
] 

Amit Sela commented on BEAM-1130:
---------------------------------

There are currently three tests reading from Kafka (also testing {{Read.Unbounded}}), and
until we have proper streaming support via triggers we rely (optimistically) on elements to
be read in the same batch.
The flakiness comes from the fact that this becomes very unpredictable on slow environments
(such as Jenkins).
By adding the possibility to manually set the bound on max records per batch (up until now
it was only via {{RateController}} feedback) I can give batches a longer read time and stop
reading (before this time ended) when I hit the max records bound.
This should do the trick though in some extreme cases of verrrry slow env. it could still
happen.
This is only putting-out fires until I provide trigger support. 

> SparkRunner ResumeFromCheckpointStreamingTest Failing
> -----------------------------------------------------
>
>                 Key: BEAM-1130
>                 URL: https://issues.apache.org/jira/browse/BEAM-1130
>             Project: Beam
>          Issue Type: Bug
>            Reporter: Jason Kuster
>            Assignee: Amit Sela
>
> Failing in (at least) suites:
> https://builds.apache.org/job/beam_PostCommit_Java_RunnableOnService_Spark/
> 381 - 385
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/
> 2063 - 2066
> Culprit change seems to be:
> https://github.com/apache/incubator-beam/commit/aba40e2de9ba058f33086eb6a913fa583a82b058
> Error:
> {code}
> Success aggregator should be greater than zero.
> Expected: not <0>
>      but: was <0>
> {code}
> Sample output:
> https://builds.apache.org/job/beam_PostCommit_Java_RunnableOnService_Spark/lastCompletedBuild/org.apache.beam$beam-runners-spark/testReport/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testRun/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message