beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Weise (JIRA)" <>
Subject [jira] [Commented] (BEAM-3261) Apex runner does not detect pipeline failure
Date Mon, 27 Nov 2017 23:12:00 GMT


Thomas Weise commented on BEAM-3261:

Yes, the test runner does not check that all assertions have succeeded. Also, if any other
error occurs, the pipeline does not "succeed", it will be stuck in retry. The test runner
caps the maximum execution time, and that may lead to false positive.

> Apex runner does not detect pipeline failure
> --------------------------------------------
>                 Key: BEAM-3261
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-apex
>            Reporter: Eugene Kirpichov
>            Assignee: Thomas Weise
>            Priority: Blocker
> I was looking at and asked myself "Don't
we already have a ValidatesRunner test for this?"
> Turns out we do:
> I ran this test with TestApexRunner, and I observed the same exception as in
, however the test passed.
> It seems that it is passing because ApexRunnerResult.waitUntilFinish() detects only PAssert
assertion errors, but not any other errors:
> This seems very problematic, as it means that 1) the status of Apex ValidatesRunner tests
cannot be trusted: if they fail in any other way other than a PAssert error, the failure will
be undetected, and 2) in general, a user can not trust a successful from Apex
> For tests in particular, some other TestXRunner's guard against such a failure mode by
verifying not only that there were no assertion failures, but also that all assertions succeeded
- using metrics: e.g.
. Fixing this would be optimal, but meanwhile, the runner should at least fail the pipeline
in case of error.

This message was sent by Atlassian JIRA

View raw message