harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [testing] Excluding tests until the problem is fixed (was: [drlvm] ... )
Date Thu, 23 Nov 2006 18:00:26 GMT


Alexei Zakharov wrote:
> Tim Ellison wrote:
>> >> >> Before you go off writing more code, just take a moment to look
at
>> >> >> HARMONY-263 and tell us what you think of it.
> 
> Geir Magnusson Jr. wrote:
>> What I meant by my comment is that I'm interested in seeing excludes in
>> external files, which can then be organized by platform, and have that
>> information used at the high-level in ant.  So for that existing JIRA,
>> reusing code for managing exclude lists is a good idea, but I'm still
>> fairly resistant to the idea of tying ourselves closer to junit.
> 
> I've looked through old HARMONY-263. In general it is close to the
> approach that was suggested by me. But George (BTW there is George?)
> proposed using rather complicated XML lists that IMHO can be not very
> handy for using in real world.
> 
> Junit part is rather small. Really, a few lines of code. The most part
> of the code (located in tests.util.SomeTest) is devoted to parsing of
> these complex XML lists by means of XPathAPI.  The most interesting
> thing here is that this code is already in our repository and has been
> lying there since March 2006.
> 
> IMHO this all is not so much about "trying ourselves closer to junit".
> This is more likely about do we like to keep the info about excludes
> in a separate file or we want to have it together with source code as
> TestNG suggests.

I'll push back against TestNG here - because having a separate file 
means that you can answer questions like "How many tests have we had to 
exclude for platform X over time?" by simply looking at the diffs for 
that file over time rather than having to through each test over time.

If things are embedded as annotations, we revert to the situation that 
we had with the DRLVM smoke tests, which was that excluding a test 
required modifying the test source, not the "control source".  I thought 
that was just awful - it meant that if I wanted to exclude on my machine 
for some reason, it meant I always had to be careful about not 
committing that modified file.

With exclude lists, we can also have things like

    make/exclude/exclude.local

which would be svn-ignored, to allow localization w/o special handling 
on commits.

geir


Mime
View raw message