harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Loenko" <mloe...@gmail.com>
Subject Re: [classlib][testing] intermittent tests
Date Tue, 27 Mar 2007 14:47:48 GMT
2007/3/27, Alexei Zakharov <alexei.zakharov@gmail.com>:
> > Do you think it is a permanent failure? :)
>
> No :)
> But my intention was at least to wait until CC sends failure alerts
> about these tests. My initial question was about a formal criteria to
> be used when placing a test to intermittent exclude list. The current
> criteria is periodical CC failure alerts, right?

I don't remember any agreement on a formal criteria.
I think that we exclude [using common sense] those tests
that may produce false alarms of CCs.

I think that the threshold (whether it fails often enough to exclude it)
should depend on the number of configurations running regularly:
if we have just on CC project that runs once a week then it's OK to
have it intermittently failing for example every 10 runs. Currently we
have dozens of CC projects and it would be good to have a false alarm
not often than once per week. Basing on the desired false alarm
frequency, number of CC projects, number of intermittently failing tests
we can derive "target" failure frequency that is enough for inclusion
to the "intermittent" exclude list :)

So...
Once per million runs is probably too rare for exclusion, but once
per 50 times is probably enough

Thanks,
Mikhail

>
> Thanks,
>
> 2007/3/27, Mikhail Loenko <mloenko@gmail.com>:
> > 2007/3/23, Alexei Zakharov <alexei.zakharov@gmail.com>:
> > > Hi all,
> > >
> > > In HARMONY-1979 we (me, Tatyana and friends) faced the problem that
> > > some tests fail very rarely. Say, one failure per 50 tries. So the
> > > question to all is: should this be treated as an intermittent failure?
> > > Thoughts?
> >
> > Do you think it is a permanent failure? :)
> >
> > if it's still a failure, we should not run the tests under CCs to avoid
> > false alarms
> >
> > Thanks,
> > Mikhail
> >
> >
> > >
> > > Thanks,
> > >
> > > 2007/3/22, tatyana doubtsova (JIRA) <jira@apache.org>:
> > > >
> > > >     [ https://issues.apache.org/jira/browse/HARMONY-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483130
]
> > > >
> > > > tatyana doubtsova commented on HARMONY-1979:
> > > > --------------------------------------------
> > > >
> > > > Alexei, the behavior of awt module tests is unstable on linux ia32, ia
64. New runs show new failures.
> > > >
> > > > lnx ia32 (gcc, debug, classlib over drlvm) intermittent failures:
> > > >
> > > > 1. java.awt.WindowTest.testSetLocationRelativeTo     fails 6 times out
of 50
> > > > expected:<java.awt.Point[x=50,y=50]> but was:<java.awt.Point[x=461,y=334]>
> > > > junit.framework.AssertionFailedError: expected:<java.awt.Point[x=50,y=50]>
but was:<java.awt.Point[x=461,y=334]> at java.awt.WindowTest.testSetLocationRelativeTo(WindowTest.java:89)
at java.lang.reflect.VMReflection.invokeMethod(VMReflection.java)
> > > >
> > > > 2. java.awt.event.InvocationEventTest   - possible crash 1 out of 50
> > > >
> > > > 3. org.apache.harmony.awt.gl.MultiRectAreaTest.testUnion - failes 2 out
of 50
> > > >     org.apache.harmony.awt.gl.MultiRectAreaTest.testSubtract - failes
2 out of 50
> > > >       java.lang.OutOfMemoryError <no stack trace available>
> > > >
> > > > 4. java.awt.font.LineBreakMeasurerTest - possible crash 1 out of 50
> > > >
> > > > 5. java.awt.font.TextHitInfoTest - possible crash 1 out of 50
> > > >
> > > > lnx ia64 (gcc, debug, classlib over drlvm) intermittent failures:
> > > >
> > > > 1. java.awt.EventDispatchThreadRTest - possible crash  4 out of 30
> > > > 2. java.awt.AccessibleAWTWindowTest  - possible crash 9 out of 30
> > > > 3. java.awt.event.InvocationEventTest - possible crash 1 out of 30
> > > > 4. java.awt.geom.FlatteningPathIteratorTest 1 - possible crash 1 out of
 30
> > > > 5. java.awt.BorderLayoutRTest - possible crash 1 out of  30
> > > > 6. java.awt.WindowTest.testSetLocationRelativeTo  fails 8 times out of
30
> > > > expected:<java.awt.Point[x=50,y=50]> but was:<java.awt.Point[x=461,y=334]>
> > > > junit.framework.AssertionFailedError: expected:<java.awt.Point[x=50,y=50]>
but was:<java.awt.Point[x=461,y=334]> at java.awt.WindowTest.testSetLocationRelativeTo(WindowTest.java:89)
at java.lang.reflect.VMReflection.invokeMethod(VMReflection.java)
> > > > 7. java.awt.KeyboardFocusManagerTest.testAddPropertyChangeListenerString
 failes 1 time out of 30
> > > > junit.framework.AssertionFailedError at java.awt.KeyboardFocusManagerTest.testAddPropertyChangeListenerString(KeyboardFocusManagerTest.java:152)
at java.lang.reflect.VMReflection.invokeMethod(VMReflection.java)
> > > >
> > > >
> > > >
> > > > > [classlib][awt][test] Many excluded AWT tests pass
> > > > > --------------------------------------------------
> > > > >
> > > > >                 Key: HARMONY-1979
> > > > >                 URL: https://issues.apache.org/jira/browse/HARMONY-1979
> > > > >             Project: Harmony
> > > > >          Issue Type: Bug
> > > > >          Components: build - test - ci
> > > > >            Reporter: Alexey A. Ivanov
> > > > >         Assigned To: Alexei Zakharov
> > > > >         Attachments: H1979-awt-unexclude-tests.patch
> > > > >
> > > > >
> > > > > Many excluded AWT tests pass successfully without any modification.
I tried both Windows and Linux.
> > > >
> > > > --
> > > > This message is automatically generated by JIRA.
> > > > -
> > > > You can reply to this email to add a comment to the issue online.
>
>
> --
> Alexei Zakharov,
> Intel ESSD
>

Mime
View raw message