Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: neutral (herse.apache.org: 216.86.168.178 is neither permitted
 nor denied by domain of geir@pobox.com)
Message-ID: <456AEA04.6040101@pobox.com>
Date: Mon, 27 Nov 2006 08:37:08 -0500
From: "Geir Magnusson Jr." <geir@pobox.com>
Reply-To: geir@pobox.com
User-Agent: Thunderbird 1.5.0.8 (Macintosh/20061025)
MIME-Version: 1.0
To: dev@harmony.apache.org
Subject: Re: [testing] test exclude list: can't we have incremental
 exclusions?
References: <8E389A5F2FEABA4CB1DEC35A25CB39CE7837AE@mssmsx411>
In-Reply-To: <8E389A5F2FEABA4CB1DEC35A25CB39CE7837AE@mssmsx411>
Content-Type: text/plain; charset=KOI8-R; format=flowed
Content-Transfer-Encoding: 8bit


Ivanov, Alexey A wrote:
>> -----Original Message-----
>> From: Geir Magnusson Jr. [mailto:geir@pobox.com]

>> Wouldn't the CI systems running on those platforms catch it automatically?
> 
> That's what we talk about, isn't it? If someone has fixed a test, it's better to exclude it from *all* the exclude lists and hoping it won't fail on all platforms. If it fails on a platform, everybody will be notified about the failure. Then we decide either to fix it, if it's easy and straightforward, or to add this test into exclude list for that certain platform.
> 
> Am I right?

I think we've gone off track.  I think we have enough degrees of freedom 
that we can test a few ways of doing this, and find out what works best. 
  I'm assuming that even if we do the "time waster" process  - remove 
from exclude lists that you can test, and then send a note to the list - 
it's not that big of a burden given we all Know for Certain :) that 
there will be very few of these, if any ;)

geir

> 
> 
> And remove a test from all exclude lists means you need to edit 8 files rather than just one.
> 
> 
> Regards,
> Alexey.
> 
> 
>> geir
>>
>>> Thanks,
>>>
>>> 24.11.06, Geir Magnusson Jr.<geir@pobox.com> �������(�):
>>>>
>>>> Alexei Zakharov wrote:
>>>>>> Could be - I don't see where the waste of time comes in.
>>>>> Just a simple arithmetic: 8 (test runs) is bigger (longer) than 1 or
>>>>> 2,  + synchronizing costs..
>>>> Sorry?  The 8 files reflect 8 *different platforms*, which are each
>>>> going to run anyway.
>>>>
>>>> geir
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>> 24.11.06, Geir Magnusson Jr.<geir@pobox.com> �������(�):
>>>>>>
>>>>>> Alexei Zakharov wrote:
>>>>>>> In this case we can run against the situation when the test will
>>>>>>> remain excluded forever for some platform.
>>>>>> Why?  Our goal is to get rid of any excluded tests.   Consider the
>>>>>> excluded tests as the existence of a regression, and we should work
>> to
>>>>>> fix ASAP.
>>>>>>
>>>>>> We aren't "parking" them there to be ignored - we're putting them
>>>> there
>>>>>> so that the build can complete while we work on the fixes.  We're in
>>>>>> essence "grandfathering-in" these regressions.
>>>>>>
>>>>>>> People have their own
>>>>>>> problems. IMHO we should do this only if there are suspicions
>>>> that the
>>>>>>> test is platform dependent. Otherwise it is a waste of time IMO
>>>> and CC
>>>>>>> can handle this.
>>>>>> Could be - I don't see where the waste of time comes in.
>>>>>>
>>>>>> geir
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> 24.11.06, Geir Magnusson Jr.<geir@pobox.com> �������(�):
>>>>>>>> Why not just remove from the platforms you have, and ask others to
>>>>>>>> update (or not) platforms you don't have as appropriate?
>>>>>>>>
>>>>>>>> geir
>>>>>>>>
>>>>>>>> Tim Ellison wrote:
>>>>>>>>> Alexei Zakharov wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>> So we just should choice what is better: to break sometimes
>>>>>> tests run
>>>>>>>>>>> or to
>>>>>>>>>>> forget enable test(s) on some platforms.
>>>>>>>>>> Yesterday, when I was removing one of the beans tests from
>>>> exclude
>>>>>>>>>> lists, I feel a bit uncomfortable while updating
>>>>>>>>>> exclude.linux.x86_64.xxx since I have no (easy) access to such
>>>>>> systems
>>>>>>>>>> and had no plans to run tests on it. IMHO (in the perfect world)
>>>>>> the
>>>>>>>>>> fact that I remove or add something from / to
>>>> exclude.linux.x86_64
>>>>>>>>>> means I've at least ran tests for this platform and obtained
>>>> some
>>>>>>>>>> result. So let's have a common list,  it's easier to deal
>>>> with it
>>>>>>>>>> psychologically.  :-)
>>>>>>>>>>
>>>>>>>>>> As for above question: +1 for being optimists, i.e. to remove
>>>>>> the test
>>>>>>>>>> from common list if it passes on all platforms available to
>>>> tester.
>>>>>>>>> +1 (otherwise I'll share the psychotherapy costs with you :-)
>>>
> 
> --
> Alexey A. Ivanov
> Intel Enterprise Solutions Software Division