harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivanov, Alexey A" <alexey.a.iva...@intel.com>
Subject RE: [testing] test exclude list: can't we have incremental exclusions?
Date Mon, 27 Nov 2006 07:51:03 GMT
>-----Original Message-----
>From: Geir Magnusson Jr. [mailto:geir@pobox.com]
>Sent: Sunday, November 26, 2006 9:23 PM
>To: dev@harmony.apache.org
>Subject: Re: [testing] test exclude list: can't we have incremental
>exclusions?
>
>
>
>Alexei Zakharov wrote:
>>> Sorry?  The 8 files reflect 8 *different platforms*, which are each
>>> going to run anyway.
>>
>> Right. You probably mean machine time here. But I was speaking about
>> *human* time. For example if we had 5 different platforms and I 've
>> ran my test only at one platform. In your scenario I need to:
>> 1. Compose message to dev@list: "guys, could you please try this test
>> on your favorite platform?" (timeWaste1)
>> 2. "platform owner" reads this (timeWaste2) and takes over the task to
>> run my test (timeWaste3)
>> 3. after (2) he/she post the ACK or failure message to dev@list
>> (timeWaste4)
>>
>> So we have four time wastes. Not a very big deal indeed. But if it
>> isn't so much necessary couldn't we avoid it at least for some obvious
>> cases?
>
>Wouldn't the CI systems running on those platforms catch it automatically?

That's what we talk about, isn't it? If someone has fixed a test, it's better to exclude it
from *all* the exclude lists and hoping it won't fail on all platforms. If it fails on a platform,
everybody will be notified about the failure. Then we decide either to fix it, if it's easy
and straightforward, or to add this test into exclude list for that certain platform.

Am I right?


And remove a test from all exclude lists means you need to edit 8 files rather than just one.


Regards,
Alexey.


>
>geir
>
>>
>> Thanks,
>>
>> 24.11.06, Geir Magnusson Jr.<geir@pobox.com> написал(а):
>>>
>>>
>>> Alexei Zakharov wrote:
>>> >> Could be - I don't see where the waste of time comes in.
>>> >
>>> > Just a simple arithmetic: 8 (test runs) is bigger (longer) than 1 or
>>> > 2,  + synchronizing costs..
>>>
>>> Sorry?  The 8 files reflect 8 *different platforms*, which are each
>>> going to run anyway.
>>>
>>> geir
>>>
>>>
>>> >
>>> > Thanks,
>>> >
>>> > 24.11.06, Geir Magnusson Jr.<geir@pobox.com> написал(а):
>>> >>
>>> >>
>>> >> Alexei Zakharov wrote:
>>> >> > In this case we can run against the situation when the test will
>>> >> > remain excluded forever for some platform.
>>> >>
>>> >> Why?  Our goal is to get rid of any excluded tests.   Consider the
>>> >> excluded tests as the existence of a regression, and we should work
>to
>>> >> fix ASAP.
>>> >>
>>> >> We aren't "parking" them there to be ignored - we're putting them
>>> there
>>> >> so that the build can complete while we work on the fixes.  We're in
>>> >> essence "grandfathering-in" these regressions.
>>> >>
>>> >> > People have their own
>>> >> > problems. IMHO we should do this only if there are suspicions
>>> that the
>>> >> > test is platform dependent. Otherwise it is a waste of time IMO
>>> and CC
>>> >> > can handle this.
>>> >>
>>> >> Could be - I don't see where the waste of time comes in.
>>> >>
>>> >> geir
>>> >>
>>> >> >
>>> >> > Thanks,
>>> >> >
>>> >> > 24.11.06, Geir Magnusson Jr.<geir@pobox.com> написал(а):
>>> >> >> Why not just remove from the platforms you have, and ask others
to
>>> >> >> update (or not) platforms you don't have as appropriate?
>>> >> >>
>>> >> >> geir
>>> >> >>
>>> >> >> Tim Ellison wrote:
>>> >> >> > Alexei Zakharov wrote:
>>> >> >> >> Hi,
>>> >> >> >>
>>> >> >> >>> So we just should choice what is better: to break
sometimes
>>> >> tests run
>>> >> >> >>> or to
>>> >> >> >>> forget enable test(s) on some platforms.
>>> >> >> >> Yesterday, when I was removing one of the beans tests
from
>>> exclude
>>> >> >> >> lists, I feel a bit uncomfortable while updating
>>> >> >> >> exclude.linux.x86_64.xxx since I have no (easy) access
to such
>>> >> systems
>>> >> >> >> and had no plans to run tests on it. IMHO (in the
perfect world)
>>> >> the
>>> >> >> >> fact that I remove or add something from / to
>>> exclude.linux.x86_64
>>> >> >> >> means I've at least ran tests for this platform and
obtained
>>> some
>>> >> >> >> result. So let's have a common list,  it's easier
to deal
>>> with it
>>> >> >> >> psychologically.  :-)
>>> >> >> >>
>>> >> >> >> As for above question: +1 for being optimists, i.e.
to remove
>>> >> the test
>>> >> >> >> from common list if it passes on all platforms available
to
>>> tester.
>>> >> >> >
>>> >> >> > +1 (otherwise I'll share the psychotherapy costs with
you :-)
>>
>>

--
Alexey A. Ivanov
Intel Enterprise Solutions Software Division

Mime
View raw message