harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [testing] Excluding tests until the problem is fixed (was: [drlvm] ... )
Date Fri, 24 Nov 2006 17:54:40 GMT


Alexei Zakharov wrote:
>> I'll push back against TestNG here - because having a separate file
>> means that you can answer questions like "How many tests have we had to
>> exclude for platform X over time?" by simply looking at the diffs for
>> that file over time rather than having to through each test over time
> 
> Well, this question should be clarified IMO. Many people think that we
> are still waiting for TestNG. We can't keep distance from everything,
> we need to choose one of the harnesses.

I'm sorry.  I wasn't clear.  I wasn't pushing back against TestNG, but 
rather the philosophy, because our multi-platform, multi-OS, multi-VM 
requirements add factors that I don't think the "put it in code" 
philosophy from TestNG needed to consider.

geir

> 
> Regards,
> 
> 23.11.06, Geir Magnusson Jr.<geir@pobox.com> написал(а):
>>
>>
>> Alexei Zakharov wrote:
>> > Tim Ellison wrote:
>> >> >> >> Before you go off writing more code, just take a moment
to 
>> look at
>> >> >> >> HARMONY-263 and tell us what you think of it.
>> >
>> > Geir Magnusson Jr. wrote:
>> >> What I meant by my comment is that I'm interested in seeing 
>> excludes in
>> >> external files, which can then be organized by platform, and have that
>> >> information used at the high-level in ant.  So for that existing JIRA,
>> >> reusing code for managing exclude lists is a good idea, but I'm still
>> >> fairly resistant to the idea of tying ourselves closer to junit.
>> >
>> > I've looked through old HARMONY-263. In general it is close to the
>> > approach that was suggested by me. But George (BTW there is George?)
>> > proposed using rather complicated XML lists that IMHO can be not very
>> > handy for using in real world.
>> >
>> > Junit part is rather small. Really, a few lines of code. The most part
>> > of the code (located in tests.util.SomeTest) is devoted to parsing of
>> > these complex XML lists by means of XPathAPI.  The most interesting
>> > thing here is that this code is already in our repository and has been
>> > lying there since March 2006.
>> >
>> > IMHO this all is not so much about "trying ourselves closer to junit".
>> > This is more likely about do we like to keep the info about excludes
>> > in a separate file or we want to have it together with source code as
>> > TestNG suggests.
>>
>> I'll push back against TestNG here - because having a separate file
>> means that you can answer questions like "How many tests have we had to
>> exclude for platform X over time?" by simply looking at the diffs for
>> that file over time rather than having to through each test over time.
>>
>> If things are embedded as annotations, we revert to the situation that
>> we had with the DRLVM smoke tests, which was that excluding a test
>> required modifying the test source, not the "control source".  I thought
>> that was just awful - it meant that if I wanted to exclude on my machine
>> for some reason, it meant I always had to be careful about not
>> committing that modified file.
>>
>> With exclude lists, we can also have things like
>>
>>     make/exclude/exclude.local
>>
>> which would be svn-ignored, to allow localization w/o special handling
>> on commits.
> 
> 

Mime
View raw message