Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (herse.apache.org: domain of mloenko@gmail.com designates
 64.233.182.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=J/kQMNztzFh55q3lo0s+aUI9pdbWkpv/6Dan459ICS/VIJyj/1uxywuzUIEHznx72+4o8qQ380dGQvhXwu6pcHXbEwlMGstCVPTvxpQIUZqJTI/aguboLYgQqBTgr21jAoiJMpFd4dprcuDFv7J+y1OGXE4IUrJSRbTQj5FgNUA=
Message-ID: <906dd82e0612230945w55830a27l61c7bf3f30b47205@mail.gmail.com>
Date: Sat, 23 Dec 2006 23:45:47 +0600
From: "Mikhail Loenko" <mloenko@gmail.com>
To: dev@harmony.apache.org
Subject: Re: [general] aiming no regression
In-Reply-To: <AC450156-2828-402D-A930-928CB9D3EC8F@pobox.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <8E389A5F2FEABA4CB1DEC35A25CB39CE88D604@mssmsx411>
	 <7273946b0612190035s2e81ac7du6f2a0584bc07bfb8@mail.gmail.com>
	 <906dd82e0612190325o5a0b4528xbfb861773d9559bf@mail.gmail.com>
	 <7273946b0612192243q1656d34bx8898a4b242ee9bce@mail.gmail.com>
	 <906dd82e0612192255w1cd98034ve25b2a0f4c093cbd@mail.gmail.com>
	 <45891419.8070301@pobox.com>
	 <7273946b0612200407v3b5a1faar55d9c6bef69f3ca@mail.gmail.com>
	 <45892E7F.5000002@pobox.com>
	 <906dd82e0612210040n6d7d85dawfe2cd73efa6505a1@mail.gmail.com>
	 <AC450156-2828-402D-A930-928CB9D3EC8F@pobox.com>

2006/12/23, Geir Magnusson Jr. <geir@pobox.com>:
>
> On Dec 21, 2006, at 3:40 AM, Mikhail Loenko wrote:
>
> > 2006/12/20, Geir Magnusson Jr. <geir@pobox.com>:
> >>
> >>
> >> Vladimir Ivanov wrote:
> >> > On 12/20/06, Geir Magnusson Jr. <geir@pobox.com> wrote:
> >> >>
> >> >>
> >> >>
> >> >> Mikhail Loenko wrote:
> >> >> > Instead we may remove all swing tests from CC when run on J9
> >> and try to
> >> >> > fix the
> >> >> > problem
> >> >>
> >> >> +1
> >> >>
> >> >> geir
> >> >
> >> >
> >> >
> >> > To remove tests from CC only we need a special exclude lists for
> >> CC. Is it
> >> > OK to store it together with modules exclude lists to update it
> >> with
> >> > classlib ws?
> >>
> >> I guess my question is "why is CC special?" IOW, shouldn't we exclude
> >> those tests for J9 in general?
> >
> > I'm +1 for special exclude lists for CC:
> >
> > if we exclude something in regular x-list, we lose that tests for
> > some time,
> > but when we exclude intermittent failures in CC we still can use
> > the tests
> > to check for stable failures.
>
> That's reasonable.  My thinking though was that we need to target
> these tests for fixing, and if they are excluded "elsewhere", we may
> not focus on them with the same determination.
>
> >
> > Example: seems like currently each swing test can intermittently fail
> > for some reason. We can exclude all swing tests from CC to have
> > reliable CC reports. But
> > if we exclude them from pre-commit testing we won't be able to test
> > any commit to swing.
>
> Yep, I agree.  Lets just be sure that we don't "forget" about the
> tests we're excluding with CC.
>
> Also, lets be careful that we don't do too much CC exclusion, because
> then we simply lose the benefit of CC :)

That's right.

We can create "intermittent" exclude list and put it next to regular
exclude lists. So, we won't forget about them.

Thanks,
Mikhail


>
> geir
>
> >
> > Thanks,
> > Mikhail
> >
> >>
> >> >
> >> >
> >> >
> >> > Also option in the build to exclude some modules will be very
> >> useful.
> >> > Something like '-Dnot.build.module=swing'.
> >> >
> >> >
> >> >
> >> > Thanks, Vladimir
> >> >
> >> >
> >> >
> >> >> >
> >> >> > Thanks,
> >> >> > Mikhail
> >> >> >
> >> >> >
> >> >> > 2006/12/20, Vladimir Ivanov <ivavladimir@gmail.com>:
> >> >> >> Actually, I was able to see these failures on swing tests
> >> only. But
> >> >> >> even for
> >> >> >> swing these failures reproduced intermittently and only when
> >> all swing
> >> >> >> tests
> >> >> >> run in the one VM.
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>  Thanks, Vladimir
> >> >> >>
> >> >> >>
> >> >> >> On 12/19/06, Mikhail Loenko <mloenko@gmail.com> wrote:
> >> >> >> >
> >> >> >> > have you seen this stack when other tests run? maybe gui
> >> >> >> > breaks something causing the failure? Are you able to
> >> reproduce the
> >> >> >> > problem?
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > Mikhail
> >> >> >> >
> >> >> >> > 2006/12/19, Vladimir Ivanov <ivavladimir@gmail.com>:
> >> >> >> > > On 12/19/06, Ivanov, Alexey A <
> >> alexey.a.ivanov@intel.com> wrote:
> >> >> >> > >
> >> >> >> > > >
> >> >> >> > > > There's only one GUI test in your list:
> >> >> >> javax.swing.JToggleButtonTest.
> >> >> >> > > > The others test text model, and this particular tests
> >> don't use
> >> >> any
> >> >> >> > > > swing UI components at all.
> >> >> >> > > >
> >> >> >> > > > If I remember your reports correctly, the latter three
> >> tests
> >> >> fail
> >> >> >> > > > because of some serialization failure.
> >> >> >> > >
> >> >> >> > >
> >> >> >> > >
> >> >> >> > > Yes, testParamString at javax.swing.JToggleButtonTest and
> >> >> >> > testSerializable
> >> >> >> > > for other tests. But actually the stack trace is similar
> >> >> (below) so
> >> >> I
> >> >> >> > think
> >> >> >> > > it not gui test problem. It is just reproduce this issue.
> >> >> >> > >
> >> >> >> > >  Thanks, Vladimir
> >> >> >> > > Stack trace:
> >> >> >> > > Test: testParamStringClass: javax.swing.JToggleButtonTest
> >> >> >> > > java.lang.ArrayIndexOutOfBoundsException
> >> >> >> > > at java.util.Arrays.mergeSort(Arrays.java:2553)
> >> >> >> > > at java.util.Arrays.mergeSort (Arrays.java:2516)
> >> >> >> > > at java.util.Arrays.sort(Arrays.java:2872)
> >> >> >> > > at java.util.Arrays.sort(Arrays.java:2889)
> >> >> >> > > at java.beans.BeanInfoWrapper.getPropertyDescriptors(
> >> >> >> > BeanInfoWrapper.java
> >> >> >> > > :77)
> >> >> >> > > at java.beans.BeanInfoWrapper.getPropertyDescriptors(
> >> >> >> > BeanInfoWrapper.java
> >> >> >> > > :74)
> >> >> >> > > at javax.swing.JComponent.paramString(JComponent.java:1334)
> >> >> >> > > at java.awt.Component.toString(Component.java:166)
> >> >> >> > > at
> >> >> >> javax.swing.JToggleButtonTest.testParamString
> >> (JToggleButtonTest.java
> >> >> >> > :64)
> >> >> >> > > at
> >> >> >> java.lang.reflect.AccessibleObject.invokeV
> >> (AccessibleObject.java:25)
> >> >> >> > > at
> >> >> >> javax.swing.BasicSwingTestCase.runBareSuper
> >> (BasicSwingTestCase.java
> >> >> :117)
> >> >> >> > > at javax.swing.SwingTestCase$1.run(SwingTestCase.java:45)
> >> >> >> > > at
> >> >> >> java.awt.event.InvocationEvent.runAndNotify
> >> (InvocationEvent.java:92)
> >> >> >> > > at
> >> >> java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:81)
> >> >> >> > > at java.awt.EventQueueCore.dispatchEventImpl
> >> (EventQueueCore.java
> >> >> :133)
> >> >> >> > > at java.awt.EventQueue.dispatchEvent(EventQueue.java:144)
> >> >> >> > > at
> >> >> >> java.awt.EventDispatchThread.runModalLoop
> >> (EventDispatchThread.java:75)
> >> >> >> > > at java.awt.EventDispatchThread.run
> >> (EventDispatchThread.java:48)
> >> >> >> > >
> >> >> >> > > Test: testSerializableClass:
> >> >> >> > > javax.swing.text.AbstractDocument_SerializationTest
> >> >> >> > > java.lang.ArrayIndexOutOfBoundsException
> >> >> >> > > at java.util.Arrays.mergeSort(Arrays.java:2553)
> >> >> >> > > at java.util.Arrays.mergeSort(Arrays.java:2516)
> >> >> >> > > at java.util.Arrays.mergeSort(Arrays.java:2517)
> >> >> >> > > at java.util.Arrays.sort(Arrays.java:2872)
> >> >> >> > > at java.util.Arrays.sort(Arrays.java:2889)
> >> >> >> > > at java.io.ObjectStreamClass.computeSerialVersionUID(
> >> >> >> > ObjectStreamClass.java
> >> >> >> > > :54)
> >> >> >> > > at
> >> >> java.io.ObjectStreamClass.addToCache(ObjectStreamClass.java:211)
> >> >> >> > > at java.io.ObjectStreamClass.lookupStreamClass(
> >> >> ObjectStreamClass.java
> >> >> >> > :937)
> >> >> >> > > at java.io.ObjectStreamClass.lookup
> >> (ObjectStreamClass.java:90)
> >> >> >> > > at java.io.ObjectStreamClass.addToCache
> >> (ObjectStreamClass.java
> >> >> :23)
> >> >> >> > > at java.io.ObjectStreamClass.lookupStreamClass(
> >> >> ObjectStreamClass.java
> >> >> >> > :937)
> >> >> >> > > at java.io.ObjectStreamClass.lookup
> >> (ObjectStreamClass.java:90)
> >> >> >> > > at java.io.ObjectOutputStream.writeClassDescForClass(
> >> >> >> > ObjectOutputStream.java
> >> >> >> > > :110)
> >> >> >> > > at java.io.ObjectOutputStream.writeNewObject(
> >> >> ObjectOutputStream.java
> >> >> >> > :1644)
> >> >> >> > > at java.io.ObjectOutputStream.writeObjectInternal(
> >> >> >> > ObjectOutputStream.java
> >> >> >> > > :1956)
> >> >> >> > > at java.io.ObjectOutputStream.writeObject
> >> >> >> (ObjectOutputStream.java:1785)
> >> >> >> > > at
> >> >> >> java.io.ObjectOutputStream.writeObject
> >> (ObjectOutputStream.java:1749)
> >> >> >> > > at javax.swing.BasicSwingTestCase.serializeObject(
> >> >> >> > BasicSwingTestCase.java
> >> >> >> > > :496)
> >> >> >> > > at javax.swing.SerializableTestCase.setUp
> >> >> >> (SerializableTestCase.java:50)
> >> >> >> > > at
> >> javax.swing.text.AbstractDocument_SerializationTest.setUp
> >> >> >> > > (AbstractDocument_SerializationTest.java:43)
> >> >> >> > >
> >> >> >> > >
> >> >> >> > > > Regards,
> >> >> >> > > > Alexey.
> >> >> >> > > >
> >> >> >> > > > >
> >> >> >> > > > >
> >> >> >> > > > >On 12/18/06, Geir Magnusson Jr. <geir@pobox.com> wrote:
> >> >> >> > > > >>
> >> >> >> > > > >>
> >> >> >> > > > >>
> >> >> >> > > > >> Mikhail Loenko wrote:
> >> >> >> > > > >> > 2006/12/18, Geir Magnusson Jr. <geir@pobox.com >:
> >> >> >> > > > >> >>
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> Mikhail Loenko wrote:
> >> >> >> > > > >> >> > 2006/12/1, Geir Magnusson Jr. <geir@pobox.com>:
> >> >> >> > > > >> >> >>
> >> >> >> > > > >> >> >>
> >> >> >> > > > >> >> >> Mikhail Loenko wrote:
> >> >> >> > > > >> >> >> > 4) We have cruise controls running
> >> classlibrary tests
> >> >> on
> >> >> >> > > > DRLVM.
> >> >> >> > > > >We
> >> >> >> > > > >> >> >> > need to decide what will we do when DRLVM
> >> +Classlib
> >> >> >> cruise
> >> >> >> > > > control
> >> >> >> > > > >> >> >> > reports failure.
> >> >> >> > > > >> >> >>
> >> >> >> > > > >> >> >> Stop and fix the problem.  Is there really a
> >> question
> >> >> >> > here?  I
> >> >> >> > > > >agree
> >> >> >> > > > >> >> >
> >> >> >> > > > >> >> > Yes, there is a question here. "Stop and fix"
> >> includes
> >> >> >> > > > "discuss".
> >> >> >> > > > >But
> >> >> >> > > > >>
> >> >> >> > > > >> >> > as we now know discussion may take several
> >> days. And
> >> >> while
> >> >> >> > some
> >> >> >> > > > >> people
> >> >> >> > > > >> >> > discuss what the problem is other people can't
> >> proceed
> >> >> with
> >> >> >> > > > >> >> > development and patch
> >> >> >> > > > >> >> > intagration.
> >> >> >> > > > >> >> >
> >> >> >> > > > >> >> > To have better pace and better CC up-time we need
> >> >> something
> >> >> >> > else
> >> >> >> > > > but
> >> >> >> > > > >> >> not
> >> >> >> > > > >> >> > just "stop and fix". I suggest "revert and
> >> continue"
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> What's the difference, other than debating the
> >> >> semantics of
> >> >> >> > "fix"
> >> >> >> > > > and
> >> >> >> > > > >> >> "revert"?
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> We all agree - but I still don't think you're
> >> clearly
> >> >> stating
> >> >> >> > the
> >> >> >> > > > >> >> problem.  I think that the core problem is that
> >> we don't
> >> >> >> > > > immediately
> >> >> >> > > > >> >> react to CC failure.
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> Immediately reacting to CC failure should be the
> >> first
> >> >> >> order of
> >> >> >> > > > the
> >> >> >> > > > >day
> >> >> >> > > > >> >> here.  Reacting to me is making the decision,
> >> quickly,
> >> >> about
> >> >> >> > > > either
> >> >> >> > > > >> >> rolling back the change ("reverting") or doing
> >> something
> >> >> >> else.
> >> >> >> > > > The
> >> >> >> > > > >key
> >> >> >> > > > >>
> >> >> >> > > > >> >> is being responsive.
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> It seems that what happens is that we wait, and
> >> then sets
> >> >> of
> >> >> >> > > > changes
> >> >> >> > > > >> >> pile up, and I think that doing mass rollbacks
> >> at that
> >> >> point
> >> >> >> > will
> >> >> >> > > > >solve
> >> >> >> > > > >> >> it, but make a mess.
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> The example of what I envision is when I broke the
> >> >> build in
> >> >> >> > DRLVM,
> >> >> >> > > > >> >> Gregory told me immediately, and I fixed
> >> immediately - w/o
> >> >> a
> >> >> >> > > > rollback.
> >> >> >> > > > >> >>
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> All I'm saying is :
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> 1) We need to be far better with reaction time
> >> >> >> > > > >> >
> >> >> >> > > > >> > I would say we need to be far better with fixing/
> >> reverting
> >> >> >> time.
> >> >> >> > > > >> > If we reacted immediately and than discussed for
> >> two weeks
> >> >> >> -- we
> >> >> >> > > > would
> >> >> >> > > > >> not
> >> >> >> > > > >> > be better than where we are now
> >> >> >> > > > >>
> >> >> >> > > > >> Yes, fixing/reverting is included. It's what I meant.
> >> >> >> > > > >>
> >> >> >> > > > >> >
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> 2) We have intelligent people - we can be agile
> >> in this by
> >> >> >> > making
> >> >> >> > > > >> >> decisions (quickly!) on a case by case basis
> >> what to do.
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> I'll also suggest that we ask each committer to
> >> check the
> >> >> CC
> >> >> >> > event
> >> >> >> > > > >> >> stream before committing, so you don't commit
> >> into a bad
> >> >> >> state
> >> >> >> > of
> >> >> >> > > > >> things.
> >> >> >> > > > >> >>
> >> >> >> > > > >> >> One of my problems is that I don't trust the CC
> >> stream,
> >> >> and
> >> >> >> > don't
> >> >> >> > > > >> >> clearly see it because it's mixed in the other
> >> drek of the
> >> >> >> > > > commits@
> >> >> >> > > > >> list.
> >> >> >> > > > >> >
> >> >> >> > > > >> > The problem is intermittent failures. I suggest
> >> that we
> >> >> >> exclude
> >> >> >> > > > >graphics
> >> >> >> > > > >> > tests
> >> >> >> > > > >> > from CCs and probably have CC-specific exclude
> >> lists for
> >> >> >> > networking
> >> >> >> > > > >> tests
> >> >> >> > > > >> > (or fix all the known intermittent failures right
> >> now :)
> >> >> >> > > > >>
> >> >> >> > > > >> good idea - works for me.
> >> >> >> > > > >>
> >> >> >> > > > >> We need to drive into stability - we've made
> >> amazing progress
> >> >> in
> >> >> >> > the
> >> >> >> > > > >> last two months, and now we're down to the really,
> >> really
> >> >> hard
> >> >> >> > stuff.
> >> >> >> > > > I
> >> >> >> > > > >> think that excluding them to get rock-solid CC
> >> reporting is
> >> >> >> step 0,
> >> >> >> > > > >> and then step 1 is try and grind out the intermittent
> >> >> failures.
> >> >> >> > > > >>
> >> >> >> > > > >> geir
> >> >> >> > > > >>
> >> >> >> > > > >>
> >> >> >> > > >
> >> >> >> > > > --
> >> >> >> > > > Alexey A. Ivanov
> >> >> >> > > > Intel Enterprise Solutions Software Division
> >> >> >> > > >
> >> >> >> > >
> >> >> >> > >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >>
> >> >
> >>
>
>