harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "tatyana doubtsova" <tatyanadoubts...@gmail.com>
Subject Re: [general] aiming no regression
Date Mon, 25 Dec 2006 09:23:30 GMT
Two last cycles of iterative classlib tests runs showed intermittent
failures of java.awt.WindowTest
See notifications on alerts@harmony.apache.org
I've filed https://issues.apache.org/jira/browse/HARMONY-2867

Thanks,
Tanya


On 25 Dec 2006 12:55:15 +0600, Egor Pasko <egor.pasko@gmail.com> wrote:
>
> On the 0x248 day of Apache Harmony Mikhail Loenko wrote:
> > 2006/12/23, Geir Magnusson Jr. <geir@pobox.com>:
> > >
> > > On Dec 21, 2006, at 3:40 AM, Mikhail Loenko wrote:
> > >
> > > > 2006/12/20, Geir Magnusson Jr. <geir@pobox.com>:
> > > >>
> > > >>
> > > >> Vladimir Ivanov wrote:
> > > >> > On 12/20/06, Geir Magnusson Jr. <geir@pobox.com> wrote:
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> Mikhail Loenko wrote:
> > > >> >> > Instead we may remove all swing tests from CC when run
on J9
> > > >> and try to
> > > >> >> > fix the
> > > >> >> > problem
> > > >> >>
> > > >> >> +1
> > > >> >>
> > > >> >> geir
> > > >> >
> > > >> >
> > > >> >
> > > >> > To remove tests from CC only we need a special exclude lists
for
> > > >> CC. Is it
> > > >> > OK to store it together with modules exclude lists to update
it
> > > >> with
> > > >> > classlib ws?
> > > >>
> > > >> I guess my question is "why is CC special?" IOW, shouldn't we
> exclude
> > > >> those tests for J9 in general?
> > > >
> > > > I'm +1 for special exclude lists for CC:
> > > >
> > > > if we exclude something in regular x-list, we lose that tests for
> > > > some time,
> > > > but when we exclude intermittent failures in CC we still can use
> > > > the tests
> > > > to check for stable failures.
> > >
> > > That's reasonable.  My thinking though was that we need to target
> > > these tests for fixing, and if they are excluded "elsewhere", we may
> > > not focus on them with the same determination.
> > >
> > > >
> > > > Example: seems like currently each swing test can intermittently
> fail
> > > > for some reason. We can exclude all swing tests from CC to have
> > > > reliable CC reports. But
> > > > if we exclude them from pre-commit testing we won't be able to test
> > > > any commit to swing.
> > >
> > > Yep, I agree.  Lets just be sure that we don't "forget" about the
> > > tests we're excluding with CC.
> > >
> > > Also, lets be careful that we don't do too much CC exclusion, because
> > > then we simply lose the benefit of CC :)
> >
> > That's right.
> >
> > We can create "intermittent" exclude list and put it next to regular
> > exclude lists. So, we won't forget about them.
>
> maintaining an exclude list is somewhat painful. We may detect
> intermittent-ness *automatically* by CC (CC reruns each failed test
> several times, excluding results if intermittent). That can be
> implemented with just 1 more option in 'ant test' (for many suites,
> ha-ha). Takes longer to run, but un-excludes intermittently failed
> tests without maintaining by hand. Howabout?
>
> > Thanks,
> > Mikhail
> >
> >
> > >
> > > geir
> > >
> > > >
> > > > Thanks,
> > > > Mikhail
> > > >
> > > >>
> > > >> >
> > > >> >
> > > >> >
> > > >> > Also option in the build to exclude some modules will be very
> > > >> useful.
> > > >> > Something like '-Dnot.build.module=swing'.
> > > >> >
> > > >> >
> > > >> >
> > > >> > Thanks, Vladimir
> > > >> >
> > > >> >
> > > >> >
> > > >> >> >
> > > >> >> > Thanks,
> > > >> >> > Mikhail
> > > >> >> >
> > > >> >> >
> > > >> >> > 2006/12/20, Vladimir Ivanov <ivavladimir@gmail.com>:
> > > >> >> >> Actually, I was able to see these failures on swing
tests
> > > >> only. But
> > > >> >> >> even for
> > > >> >> >> swing these failures reproduced intermittently and
only when
> > > >> all swing
> > > >> >> >> tests
> > > >> >> >> run in the one VM.
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>  Thanks, Vladimir
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> On 12/19/06, Mikhail Loenko <mloenko@gmail.com>
wrote:
> > > >> >> >> >
> > > >> >> >> > have you seen this stack when other tests run?
maybe gui
> > > >> >> >> > breaks something causing the failure? Are you
able to
> > > >> reproduce the
> > > >> >> >> > problem?
> > > >> >> >> >
> > > >> >> >> > Thanks,
> > > >> >> >> > Mikhail
> > > >> >> >> >
> > > >> >> >> > 2006/12/19, Vladimir Ivanov <ivavladimir@gmail.com>:
> > > >> >> >> > > On 12/19/06, Ivanov, Alexey A <
> > > >> alexey.a.ivanov@intel.com> wrote:
> > > >> >> >> > >
> > > >> >> >> > > >
> > > >> >> >> > > > There's only one GUI test in your
list:
> > > >> >> >> javax.swing.JToggleButtonTest.
> > > >> >> >> > > > The others test text model, and this
particular tests
> > > >> don't use
> > > >> >> any
> > > >> >> >> > > > swing UI components at all.
> > > >> >> >> > > >
> > > >> >> >> > > > If I remember your reports correctly,
the latter three
> > > >> tests
> > > >> >> fail
> > > >> >> >> > > > because of some serialization failure.
> > > >> >> >> > >
> > > >> >> >> > >
> > > >> >> >> > >
> > > >> >> >> > > Yes, testParamString at javax.swing.JToggleButtonTest
and
> > > >> >> >> > testSerializable
> > > >> >> >> > > for other tests. But actually the stack
trace is similar
> > > >> >> (below) so
> > > >> >> I
> > > >> >> >> > think
> > > >> >> >> > > it not gui test problem. It is just reproduce
this issue.
> > > >> >> >> > >
> > > >> >> >> > >  Thanks, Vladimir
> > > >> >> >> > > Stack trace:
> > > >> >> >> > > Test: testParamStringClass: javax.swing.JToggleButtonTest
> > > >> >> >> > > java.lang.ArrayIndexOutOfBoundsException
> > > >> >> >> > > at java.util.Arrays.mergeSort(Arrays.java:2553)
> > > >> >> >> > > at java.util.Arrays.mergeSort (Arrays.java:2516)
> > > >> >> >> > > at java.util.Arrays.sort(Arrays.java:2872)
> > > >> >> >> > > at java.util.Arrays.sort(Arrays.java:2889)
> > > >> >> >> > > at java.beans.BeanInfoWrapper.getPropertyDescriptors(
> > > >> >> >> > BeanInfoWrapper.java
> > > >> >> >> > > :77)
> > > >> >> >> > > at java.beans.BeanInfoWrapper.getPropertyDescriptors(
> > > >> >> >> > BeanInfoWrapper.java
> > > >> >> >> > > :74)
> > > >> >> >> > > at javax.swing.JComponent.paramString(JComponent.java
> :1334)
> > > >> >> >> > > at java.awt.Component.toString(Component.java:166)
> > > >> >> >> > > at
> > > >> >> >> javax.swing.JToggleButtonTest.testParamString
> > > >> (JToggleButtonTest.java
> > > >> >> >> > :64)
> > > >> >> >> > > at
> > > >> >> >> java.lang.reflect.AccessibleObject.invokeV
> > > >> (AccessibleObject.java:25)
> > > >> >> >> > > at
> > > >> >> >> javax.swing.BasicSwingTestCase.runBareSuper
> > > >> (BasicSwingTestCase.java
> > > >> >> :117)
> > > >> >> >> > > at javax.swing.SwingTestCase$1.run(SwingTestCase.java:45)
> > > >> >> >> > > at
> > > >> >> >> java.awt.event.InvocationEvent.runAndNotify
> > > >> (InvocationEvent.java:92)
> > > >> >> >> > > at
> > > >> >> java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:81)
> > > >> >> >> > > at java.awt.EventQueueCore.dispatchEventImpl
> > > >> (EventQueueCore.java
> > > >> >> :133)
> > > >> >> >> > > at java.awt.EventQueue.dispatchEvent(EventQueue.java:144)
> > > >> >> >> > > at
> > > >> >> >> java.awt.EventDispatchThread.runModalLoop
> > > >> (EventDispatchThread.java:75)
> > > >> >> >> > > at java.awt.EventDispatchThread.run
> > > >> (EventDispatchThread.java:48)
> > > >> >> >> > >
> > > >> >> >> > > Test: testSerializableClass:
> > > >> >> >> > > javax.swing.text.AbstractDocument_SerializationTest
> > > >> >> >> > > java.lang.ArrayIndexOutOfBoundsException
> > > >> >> >> > > at java.util.Arrays.mergeSort(Arrays.java:2553)
> > > >> >> >> > > at java.util.Arrays.mergeSort(Arrays.java:2516)
> > > >> >> >> > > at java.util.Arrays.mergeSort(Arrays.java:2517)
> > > >> >> >> > > at java.util.Arrays.sort(Arrays.java:2872)
> > > >> >> >> > > at java.util.Arrays.sort(Arrays.java:2889)
> > > >> >> >> > > at java.io.ObjectStreamClass.computeSerialVersionUID(
> > > >> >> >> > ObjectStreamClass.java
> > > >> >> >> > > :54)
> > > >> >> >> > > at
> > > >> >> java.io.ObjectStreamClass.addToCache(ObjectStreamClass.java:211)
> > > >> >> >> > > at java.io.ObjectStreamClass.lookupStreamClass(
> > > >> >> ObjectStreamClass.java
> > > >> >> >> > :937)
> > > >> >> >> > > at java.io.ObjectStreamClass.lookup
> > > >> (ObjectStreamClass.java:90)
> > > >> >> >> > > at java.io.ObjectStreamClass.addToCache
> > > >> (ObjectStreamClass.java
> > > >> >> :23)
> > > >> >> >> > > at java.io.ObjectStreamClass.lookupStreamClass(
> > > >> >> ObjectStreamClass.java
> > > >> >> >> > :937)
> > > >> >> >> > > at java.io.ObjectStreamClass.lookup
> > > >> (ObjectStreamClass.java:90)
> > > >> >> >> > > at java.io.ObjectOutputStream.writeClassDescForClass(
> > > >> >> >> > ObjectOutputStream.java
> > > >> >> >> > > :110)
> > > >> >> >> > > at java.io.ObjectOutputStream.writeNewObject(
> > > >> >> ObjectOutputStream.java
> > > >> >> >> > :1644)
> > > >> >> >> > > at java.io.ObjectOutputStream.writeObjectInternal(
> > > >> >> >> > ObjectOutputStream.java
> > > >> >> >> > > :1956)
> > > >> >> >> > > at java.io.ObjectOutputStream.writeObject
> > > >> >> >> (ObjectOutputStream.java:1785)
> > > >> >> >> > > at
> > > >> >> >> java.io.ObjectOutputStream.writeObject
> > > >> (ObjectOutputStream.java:1749)
> > > >> >> >> > > at javax.swing.BasicSwingTestCase.serializeObject(
> > > >> >> >> > BasicSwingTestCase.java
> > > >> >> >> > > :496)
> > > >> >> >> > > at javax.swing.SerializableTestCase.setUp
> > > >> >> >> (SerializableTestCase.java:50)
> > > >> >> >> > > at
> > > >> javax.swing.text.AbstractDocument_SerializationTest.setUp
> > > >> >> >> > > (AbstractDocument_SerializationTest.java:43)
> > > >> >> >> > >
> > > >> >> >> > >
> > > >> >> >> > > > Regards,
> > > >> >> >> > > > Alexey.
> > > >> >> >> > > >
> > > >> >> >> > > > >
> > > >> >> >> > > > >
> > > >> >> >> > > > >On 12/18/06, Geir Magnusson Jr.
<geir@pobox.com>
> wrote:
> > > >> >> >> > > > >>
> > > >> >> >> > > > >>
> > > >> >> >> > > > >>
> > > >> >> >> > > > >> Mikhail Loenko wrote:
> > > >> >> >> > > > >> > 2006/12/18, Geir Magnusson
Jr. <geir@pobox.com >:
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> Mikhail Loenko
wrote:
> > > >> >> >> > > > >> >> > 2006/12/1,
Geir Magnusson Jr. <geir@pobox.com>:
> > > >> >> >> > > > >> >> >>
> > > >> >> >> > > > >> >> >>
> > > >> >> >> > > > >> >> >> Mikhail
Loenko wrote:
> > > >> >> >> > > > >> >> >> > 4)
We have cruise controls running
> > > >> classlibrary tests
> > > >> >> on
> > > >> >> >> > > > DRLVM.
> > > >> >> >> > > > >We
> > > >> >> >> > > > >> >> >> > need
to decide what will we do when DRLVM
> > > >> +Classlib
> > > >> >> >> cruise
> > > >> >> >> > > > control
> > > >> >> >> > > > >> >> >> > reports
failure.
> > > >> >> >> > > > >> >> >>
> > > >> >> >> > > > >> >> >> Stop and
fix the problem.  Is there really a
> > > >> question
> > > >> >> >> > here?  I
> > > >> >> >> > > > >agree
> > > >> >> >> > > > >> >> >
> > > >> >> >> > > > >> >> > Yes, there
is a question here. "Stop and fix"
> > > >> includes
> > > >> >> >> > > > "discuss".
> > > >> >> >> > > > >But
> > > >> >> >> > > > >>
> > > >> >> >> > > > >> >> > as we now
know discussion may take several
> > > >> days. And
> > > >> >> while
> > > >> >> >> > some
> > > >> >> >> > > > >> people
> > > >> >> >> > > > >> >> > discuss what
the problem is other people can't
> > > >> proceed
> > > >> >> with
> > > >> >> >> > > > >> >> > development
and patch
> > > >> >> >> > > > >> >> > intagration.
> > > >> >> >> > > > >> >> >
> > > >> >> >> > > > >> >> > To have better
pace and better CC up-time we
> need
> > > >> >> something
> > > >> >> >> > else
> > > >> >> >> > > > but
> > > >> >> >> > > > >> >> not
> > > >> >> >> > > > >> >> > just "stop
and fix". I suggest "revert and
> > > >> continue"
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> What's the difference,
other than debating the
> > > >> >> semantics of
> > > >> >> >> > "fix"
> > > >> >> >> > > > and
> > > >> >> >> > > > >> >> "revert"?
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> We all agree -
but I still don't think you're
> > > >> clearly
> > > >> >> stating
> > > >> >> >> > the
> > > >> >> >> > > > >> >> problem.  I think
that the core problem is that
> > > >> we don't
> > > >> >> >> > > > immediately
> > > >> >> >> > > > >> >> react to CC failure.
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> Immediately reacting
to CC failure should be the
> > > >> first
> > > >> >> >> order of
> > > >> >> >> > > > the
> > > >> >> >> > > > >day
> > > >> >> >> > > > >> >> here.  Reacting
to me is making the decision,
> > > >> quickly,
> > > >> >> about
> > > >> >> >> > > > either
> > > >> >> >> > > > >> >> rolling back the
change ("reverting") or doing
> > > >> something
> > > >> >> >> else.
> > > >> >> >> > > > The
> > > >> >> >> > > > >key
> > > >> >> >> > > > >>
> > > >> >> >> > > > >> >> is being responsive.
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> It seems that what
happens is that we wait, and
> > > >> then sets
> > > >> >> of
> > > >> >> >> > > > changes
> > > >> >> >> > > > >> >> pile up, and I
think that doing mass rollbacks
> > > >> at that
> > > >> >> point
> > > >> >> >> > will
> > > >> >> >> > > > >solve
> > > >> >> >> > > > >> >> it, but make a
mess.
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> The example of
what I envision is when I broke
> the
> > > >> >> build in
> > > >> >> >> > DRLVM,
> > > >> >> >> > > > >> >> Gregory told me
immediately, and I fixed
> > > >> immediately - w/o
> > > >> >> a
> > > >> >> >> > > > rollback.
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> All I'm saying
is :
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> 1) We need to be
far better with reaction time
> > > >> >> >> > > > >> >
> > > >> >> >> > > > >> > I would say we need
to be far better with fixing/
> > > >> reverting
> > > >> >> >> time.
> > > >> >> >> > > > >> > If we reacted immediately
and than discussed for
> > > >> two weeks
> > > >> >> >> -- we
> > > >> >> >> > > > would
> > > >> >> >> > > > >> not
> > > >> >> >> > > > >> > be better than where
we are now
> > > >> >> >> > > > >>
> > > >> >> >> > > > >> Yes, fixing/reverting is
included. It's what I
> meant.
> > > >> >> >> > > > >>
> > > >> >> >> > > > >> >
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> 2) We have intelligent
people - we can be agile
> > > >> in this by
> > > >> >> >> > making
> > > >> >> >> > > > >> >> decisions (quickly!)
on a case by case basis
> > > >> what to do.
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> I'll also suggest
that we ask each committer to
> > > >> check the
> > > >> >> CC
> > > >> >> >> > event
> > > >> >> >> > > > >> >> stream before committing,
so you don't commit
> > > >> into a bad
> > > >> >> >> state
> > > >> >> >> > of
> > > >> >> >> > > > >> things.
> > > >> >> >> > > > >> >>
> > > >> >> >> > > > >> >> One of my problems
is that I don't trust the CC
> > > >> stream,
> > > >> >> and
> > > >> >> >> > don't
> > > >> >> >> > > > >> >> clearly see it
because it's mixed in the other
> > > >> drek of the
> > > >> >> >> > > > commits@
> > > >> >> >> > > > >> list.
> > > >> >> >> > > > >> >
> > > >> >> >> > > > >> > The problem is intermittent
failures. I suggest
> > > >> that we
> > > >> >> >> exclude
> > > >> >> >> > > > >graphics
> > > >> >> >> > > > >> > tests
> > > >> >> >> > > > >> > from CCs and probably
have CC-specific exclude
> > > >> lists for
> > > >> >> >> > networking
> > > >> >> >> > > > >> tests
> > > >> >> >> > > > >> > (or fix all the known
intermittent failures right
> > > >> now :)
> > > >> >> >> > > > >>
> > > >> >> >> > > > >> good idea - works for me.
> > > >> >> >> > > > >>
> > > >> >> >> > > > >> We need to drive into stability
- we've made
> > > >> amazing progress
> > > >> >> in
> > > >> >> >> > the
> > > >> >> >> > > > >> last two months, and now
we're down to the really,
> > > >> really
> > > >> >> hard
> > > >> >> >> > stuff.
> > > >> >> >> > > > I
> > > >> >> >> > > > >> think that excluding them
to get rock-solid CC
> > > >> reporting is
> > > >> >> >> step 0,
> > > >> >> >> > > > >> and then step 1 is try and
grind out the
> intermittent
> > > >> >> failures.
> > > >> >> >> > > > >>
> > > >> >> >> > > > >> geir
> > > >> >> >> > > > >>
> > > >> >> >> > > > >>
> > > >> >> >> > > >
> > > >> >> >> > > > --
> > > >> >> >> > > > Alexey A. Ivanov
> > > >> >> >> > > > Intel Enterprise Solutions Software
Division
> > > >> >> >> > > >
> > > >> >> >> > >
> > > >> >> >> > >
> > > >> >> >> >
> > > >> >> >>
> > > >> >> >>
> > > >> >>
> > > >> >
> > > >>
> > >
> > >
> >
>
> --
> Egor Pasko
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message