brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aled Sage (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (BROOKLYN-272) Non-deterministic test failures in jenkins
Date Fri, 20 May 2016 11:19:13 GMT

     [ https://issues.apache.org/jira/browse/BROOKLYN-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aled Sage updated BROOKLYN-272:
-------------------------------
    Description: 
There are several non-deterministic tests that fail occasionally in the apache jenkins build.
We should fix those tests: either the underlying bug if there is one, or the way to the test
has been written to make it more deterministic.

Having build failures unrelated to the changes being made in a PR (and on master) is extremely
disruptive for development. We must avoid that wherever possible.

I therefore suggest we disable these non-deterministic failing tests immediately, and use
this jira issue to track fixing them. We can annotate the tests with {{@Test(groups={"WIP",
"Non-deterministic-failure"})}} to make them easy to find. (note the "Non-deterministic-failure"
group is not used by any profiles, it's just there for searching purposes).

The tests I have identified from recent build failures are:

* {{BasicStartableTest.testTransitionsThroughLifecycles}} - see https://issues.apache.org/jira/browse/BROOKLYN-256

* {{AbstractGeoDnsServiceTest.testFiltersForRunningEntities}}, failing with:

{noformat}
    org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed succeeds-eventually,
69 attempts, 30003ms elapsed: AssertionError: val={R48yHLqg=<address-ignored>, eBHFc4Qd=<address-ignored>}
            at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
            at org.apache.brooklyn.test.Asserts.assertTrue(Asserts.java:703)
            at org.apache.brooklyn.core.entity.EntityAsserts$2.run(EntityAsserts.java:92)
            at org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
            at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
            at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
            at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:89)
            at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:84)
            at org.apache.brooklyn.entity.dns.AbstractGeoDnsServiceTest.testFiltersForRunningEntities(AbstractGeoDnsServiceTest.java:263)
{noformat}

* {{LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers}}, failing with:

{noformat}
    org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed succeeds-eventually,
29 attempts, 10002ms elapsed: AssertionError: actual=[18.0, 21.0, 19.0, 0.0, 20.0, 0.0, 20.0,
36.0, 19.0, 0.0, 19.0, 21.0, 21.0, 19.0, 21.0, 36.0, 21.0, 19.0, 18.0, 36.0]; expected=[20.0,
20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0,
20.0, 20.0, 20.0, 20.0] expected [20.0] but found [0.0]
            at org.testng.Assert.fail(Assert.java:94)
            at org.testng.Assert.failNotEquals(Assert.java:494)
            at org.testng.Assert.assertEquals(Assert.java:207)
            at org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkrates(AbstractLoadBalancingPolicyTest.java:122)
            at org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest$2.run(AbstractLoadBalancingPolicyTest.java:138)
            at org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
            at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
            at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
            at org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkratesEventually(AbstractLoadBalancingPolicyTest.java:136)
            at org.apache.brooklyn.policy.loadbalancing.LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers(LoadBalancingPolicyConcurrencyTest.java:101)
{noformat}

* {{PollerTest.testFeedContinuesWhenPollerThrows}}, failing with:

{noformat}
    testFeedContinuesWhenPollerThrows(org.apache.brooklyn.core.feed.PollerTest)  Time elapsed:
3.556 sec  <<< FAILURE!
    org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed succeeds-eventually,
4 attempts, 105ms elapsed: AssertionError: entity=FeedExceptionEntityImpl{id=dYG0SKbP}; attribute=Sensor:
flag (java.lang.Boolean) expected [false] but found [true]
        at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
        at org.apache.brooklyn.test.Asserts.failNotEquals(Asserts.java:114)
        at org.apache.brooklyn.test.Asserts.assertEquals(Asserts.java:436)
        at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEquals(EntityAsserts.java:54)
        at org.apache.brooklyn.core.entity.EntityAsserts$1.run(EntityAsserts.java:70)
        at org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
        at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
        at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
        at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEqualsEventually(EntityAsserts.java:67)
        at org.apache.brooklyn.core.feed.PollerTest.testFeedContinuesWhenPollerThrows(PollerTest.java:65)
{noformat}

  was:
There are several non-deterministic tests that fail occasionally in the apache jenkins build.
We should fix those tests: either the underlying bug if there is one, or the way to the test
has been written to make it more deterministic.

Having build failures unrelated to the changes being made in a PR (and on master) is extremely
disruptive for development. We must avoid that wherever possible.

I therefore suggest we disable these non-deterministic failing tests immediately, and use
this jira issue to track fixing them. We can annotate the tests with {{@Test(groups={"WIP",
"Non-deterministic-failure"})} to make them easy to find. (note the "Non-deterministic-failure"
group is not used by any profiles, it's just there for searching purposes).

The tests I have identified from recent build failures are:

* {{BasicStartableTest.testTransitionsThroughLifecycles}} - see https://issues.apache.org/jira/browse/BROOKLYN-256

* {{AbstractGeoDnsServiceTest.testFiltersForRunningEntities}}, failing with:

{noformat}
    org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed succeeds-eventually,
69 attempts, 30003ms elapsed: AssertionError: val={R48yHLqg=<address-ignored>, eBHFc4Qd=<address-ignored>}
            at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
            at org.apache.brooklyn.test.Asserts.assertTrue(Asserts.java:703)
            at org.apache.brooklyn.core.entity.EntityAsserts$2.run(EntityAsserts.java:92)
            at org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
            at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
            at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
            at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:89)
            at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:84)
            at org.apache.brooklyn.entity.dns.AbstractGeoDnsServiceTest.testFiltersForRunningEntities(AbstractGeoDnsServiceTest.java:263)
{noformat}

* {{LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers}}, failing with:

{noformat}
    org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed succeeds-eventually,
29 attempts, 10002ms elapsed: AssertionError: actual=[18.0, 21.0, 19.0, 0.0, 20.0, 0.0, 20.0,
36.0, 19.0, 0.0, 19.0, 21.0, 21.0, 19.0, 21.0, 36.0, 21.0, 19.0, 18.0, 36.0]; expected=[20.0,
20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0,
20.0, 20.0, 20.0, 20.0] expected [20.0] but found [0.0]
            at org.testng.Assert.fail(Assert.java:94)
            at org.testng.Assert.failNotEquals(Assert.java:494)
            at org.testng.Assert.assertEquals(Assert.java:207)
            at org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkrates(AbstractLoadBalancingPolicyTest.java:122)
            at org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest$2.run(AbstractLoadBalancingPolicyTest.java:138)
            at org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
            at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
            at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
            at org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkratesEventually(AbstractLoadBalancingPolicyTest.java:136)
            at org.apache.brooklyn.policy.loadbalancing.LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers(LoadBalancingPolicyConcurrencyTest.java:101)
{noformat}

* {{PollerTest.testFeedContinuesWhenPollerThrows}}, failing with:

{noformat}
    testFeedContinuesWhenPollerThrows(org.apache.brooklyn.core.feed.PollerTest)  Time elapsed:
3.556 sec  <<< FAILURE!
    org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed succeeds-eventually,
4 attempts, 105ms elapsed: AssertionError: entity=FeedExceptionEntityImpl{id=dYG0SKbP}; attribute=Sensor:
flag (java.lang.Boolean) expected [false] but found [true]
        at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
        at org.apache.brooklyn.test.Asserts.failNotEquals(Asserts.java:114)
        at org.apache.brooklyn.test.Asserts.assertEquals(Asserts.java:436)
        at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEquals(EntityAsserts.java:54)
        at org.apache.brooklyn.core.entity.EntityAsserts$1.run(EntityAsserts.java:70)
        at org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
        at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
        at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
        at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEqualsEventually(EntityAsserts.java:67)
        at org.apache.brooklyn.core.feed.PollerTest.testFeedContinuesWhenPollerThrows(PollerTest.java:65)
{noformat}


> Non-deterministic test failures in jenkins
> ------------------------------------------
>
>                 Key: BROOKLYN-272
>                 URL: https://issues.apache.org/jira/browse/BROOKLYN-272
>             Project: Brooklyn
>          Issue Type: Bug
>            Reporter: Aled Sage
>
> There are several non-deterministic tests that fail occasionally in the apache jenkins
build. We should fix those tests: either the underlying bug if there is one, or the way to
the test has been written to make it more deterministic.
> Having build failures unrelated to the changes being made in a PR (and on master) is
extremely disruptive for development. We must avoid that wherever possible.
> I therefore suggest we disable these non-deterministic failing tests immediately, and
use this jira issue to track fixing them. We can annotate the tests with {{@Test(groups={"WIP",
"Non-deterministic-failure"})}} to make them easy to find. (note the "Non-deterministic-failure"
group is not used by any profiles, it's just there for searching purposes).
> The tests I have identified from recent build failures are:
> * {{BasicStartableTest.testTransitionsThroughLifecycles}} - see https://issues.apache.org/jira/browse/BROOKLYN-256
> * {{AbstractGeoDnsServiceTest.testFiltersForRunningEntities}}, failing with:
> {noformat}
>     org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed succeeds-eventually,
69 attempts, 30003ms elapsed: AssertionError: val={R48yHLqg=<address-ignored>, eBHFc4Qd=<address-ignored>}
>             at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
>             at org.apache.brooklyn.test.Asserts.assertTrue(Asserts.java:703)
>             at org.apache.brooklyn.core.entity.EntityAsserts$2.run(EntityAsserts.java:92)
>             at org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
>             at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
>             at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
>             at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:89)
>             at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:84)
>             at org.apache.brooklyn.entity.dns.AbstractGeoDnsServiceTest.testFiltersForRunningEntities(AbstractGeoDnsServiceTest.java:263)
> {noformat}
> * {{LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers}}, failing with:
> {noformat}
>     org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed succeeds-eventually,
29 attempts, 10002ms elapsed: AssertionError: actual=[18.0, 21.0, 19.0, 0.0, 20.0, 0.0, 20.0,
36.0, 19.0, 0.0, 19.0, 21.0, 21.0, 19.0, 21.0, 36.0, 21.0, 19.0, 18.0, 36.0]; expected=[20.0,
20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0,
20.0, 20.0, 20.0, 20.0] expected [20.0] but found [0.0]
>             at org.testng.Assert.fail(Assert.java:94)
>             at org.testng.Assert.failNotEquals(Assert.java:494)
>             at org.testng.Assert.assertEquals(Assert.java:207)
>             at org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkrates(AbstractLoadBalancingPolicyTest.java:122)
>             at org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest$2.run(AbstractLoadBalancingPolicyTest.java:138)
>             at org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
>             at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
>             at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
>             at org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkratesEventually(AbstractLoadBalancingPolicyTest.java:136)
>             at org.apache.brooklyn.policy.loadbalancing.LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers(LoadBalancingPolicyConcurrencyTest.java:101)
> {noformat}
> * {{PollerTest.testFeedContinuesWhenPollerThrows}}, failing with:
> {noformat}
>     testFeedContinuesWhenPollerThrows(org.apache.brooklyn.core.feed.PollerTest)  Time
elapsed: 3.556 sec  <<< FAILURE!
>     org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed succeeds-eventually,
4 attempts, 105ms elapsed: AssertionError: entity=FeedExceptionEntityImpl{id=dYG0SKbP}; attribute=Sensor:
flag (java.lang.Boolean) expected [false] but found [true]
>         at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
>         at org.apache.brooklyn.test.Asserts.failNotEquals(Asserts.java:114)
>         at org.apache.brooklyn.test.Asserts.assertEquals(Asserts.java:436)
>         at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEquals(EntityAsserts.java:54)
>         at org.apache.brooklyn.core.entity.EntityAsserts$1.run(EntityAsserts.java:70)
>         at org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
>         at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
>         at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
>         at org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEqualsEventually(EntityAsserts.java:67)
>         at org.apache.brooklyn.core.feed.PollerTest.testFeedContinuesWhenPollerThrows(PollerTest.java:65)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message