Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: neutral (herse.apache.org: 216.86.168.178 is neither permitted
 nor denied by domain of geir@pobox.com)
Message-ID: <455CFCB5.7050609@pobox.com>
Date: Thu, 16 Nov 2006 19:05:09 -0500
From: "Geir Magnusson Jr." <geir@pobox.com>
Reply-To: geir@pobox.com
User-Agent: Thunderbird 1.5.0.8 (Macintosh/20061025)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [drlvm][unit] 100% of class library tests pass
References: <8E389A5F2FEABA4CB1DEC35A25CB39CE73C8D6@mssmsx411>
	 <469bff730611160240y2c6176deu62fbfaeb53142811@mail.gmail.com>
 <c30d70960611161453m4b5855f1jb86b60c5ad2898ea@mail.gmail.com>
In-Reply-To: <c30d70960611161453m4b5855f1jb86b60c5ad2898ea@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


Alexei Fedotov wrote:
> Pavel,
> The life started showing that you were correct. Today there were no
> report on http://harmonytest.org. Even if I would like to be a living
> notification, I couldn't.
> 
> Vladimir,
> The thing which concerns me most is not an absence of results - I
> believe this is just a technical problem. For me the main problem is
> that without you there is no way to understand what happens.

I don't understand that.

The goal here is to establish the build-test framework as the thing that 
everyone uses- we aren't dependent only upon Vladimir.

I'll have a version running on 64-bit ubuntu whenever it works, and 
probalby 32-bit as well reporting in...

> 
> Can we made a process of reporting more open? For example, can we tune
> CC to send a notification on upload status? If there is a successful
> notification, then the file is uploaded successfully, and we need to
> ask Anton why it is not visible. Otherwise we'll know the problem is
> in CC.

Um.  I'd prefer if we didn't spam the lists with every good result - we 
only want to know when there's breakage or "fixage".

geir

> 
> Thanks!
> 
> 
> 
> On 11/16/06, Pavel Ozhdikhin <pavel.ozhdikhin@gmail.com> wrote:
>> Sorry to say but it actually does not work until there is no 
>> notifications
>> to the mailing list and no immediate reaction to the regressions.
>>
>> thanks,
>> Pavel
>>
>>
>> On 11/16/06, Fedotov, Alexei A <alexei.a.fedotov@intel.com> wrote:
>> >
>> > Pavel, All,
>> >
>> > Let me add one "pro" for the second approach: it works already.
>> > Vladimir's scripts daily upload test results to http://harmonytest.org.
>> >
>> > With best regards,
>> > Alexei Fedotov,
>> > Intel Java & XML Engineering
>> >
>> > >-----Original Message-----
>> > >From: Tim Ellison [mailto:t.p.ellison@gmail.com]
>> > >Sent: Thursday, November 16, 2006 12:37 PM
>> > >To: harmony-dev@incubator.apache.org
>> > >Subject: Re: [drlvm][unit] 100% of class library tests pass
>> > >
>> > >Pavel Ozhdikhin wrote:
>> > >> We have to evolving systems - classlib and DRLVM. To check 
>> commits to
>> > >> classlib we need a stable DRLVM which can pass 100% of HUT. 
>> Otherwise
>> > >it's
>> > >> impossible to use DRLVM for pre-commit testing - you never know
>> > whether
>> > >> your
>> > >> test fail because of your patch or due to latest changes in DRLVM.
>> > >>
>> > >> I remember the time when DRLVM and Jitrino actively evolved - for
>> > some
>> > >time
>> > >> JIT had to use an older version of DRLVM which could pass all commit
>> > >> criteria because newer versions suffered from regressions. And
>> > finally we
>> > >> came to comon strict commit criteria which prevented regressions in
>> > both
>> > >VM
>> > >> and JIT.
>> > >>
>> > >> To avoid regressions using DRLVM in classlib testing I see 3 
>> possible
>> > >> solutions:
>> > >>
>> > >> 1. Use one fixed DRLVM version which can pass 100% HUT test. Update
>> > this
>> > >> version from time to time.
>> > >>    Pros: + Less time to run DRLVM pre-commit tests
>> > >>              + Classlib does not suffer from regressions in DRLVM
>> > >>    Cons: - DRLVM will suffer from regressions
>> > >>               - Classlib can not use the latest DRLVM
>> > >>               - Need additional efforts to regain stability on DRLVM
>> > >>                 when we want to update the version for classlib
>> > testing
>> > >>
>> > >> 2. Add HUT to CruiseControl testing on DRLVM and rollback commits
>> > causing
>> > >> regressions
>> > >>    Pros: + Less time to run DRLVM pre-commit tests
>> > >>              + Classlib can use the latest DRLVM
>> > >>    Cons: - Classlib can suffer from DRLVM regressions (time lag
>> > before
>> > >> rollback)
>> > >>               - It is not always clear which commit caused a
>> > regression
>> > >>               - Rollbacks are costly
>> > >>
>> > >> 3. Add HUT to the commit criteria for DRLVM
>> > >>     Pros: + Classlib always can use the latest DRLVM
>> > >>               + DRLVM has no regressions regarding to HUT
>> > >>     Cons: - More time to run DRLVM pre-commit tests (I was told that
>> > HUT
>> > >>                   take 25 minutes running in single JVM mode)
>> > >>
>> > >> I think that preventing a problem is better than solving it
>> > afterwards.
>> > >So,
>> > >> I personally would choose the 3rd approach, don't mind against the
>> > second
>> > >> and dislike the first one. Probably some combination of these is
>> > >possible.
>> > >
>> > >While I appreciate the desire to keep things stable, I think it is
>> > >unreasonable to ask developers to run the entire test suite each time.
>> > >As we have seen in the classlib code, running targeted tests before
>> > >commit and leaving the build machines to run continuous tests ensures
>> > >that we are productive and are notified of breakages in time to easily
>> > >back out a patch and re-evaluate.
>> > >
>> > >With the amount of machine time we have running harmony tests on
>> > >different cpu's/os's/compilers/etc we are getting better coverage than
>> > >any individual could be expected to provide.
>> > >
>> > >Which is a long way of saying I think option (2) above is best -- and
>> > >relies on the bid machines letting us know if things break, and the
>> > >commitment from all of us to go straight in and fix it.
>> > >
>> > >Regards,
>> > >Tim
>> > >
>> > >--
>> > >
>> > >Tim Ellison (t.p.ellison@gmail.com)
>> > >IBM Java technology centre, UK.
>> >
>>
>>
> 
>