harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Zakharov" <alexei.zakha...@gmail.com>
Subject Re: [testing] Peace (was: Re: svn commit: r419522 - in /incubator/harmony/enhanced/classlib/trunk/modules/nio: .classpath build.xml make/hyproperties.xml src/test/java/common/ src/test/java/common/org/ src/test/java/linux/ src/test/java/org/ src/test
Date Mon, 24 Jul 2006 15:25:43 GMT
Hi Vladimir,

> Cons:

> 1. More complex algorithms of grouping / selecting tests for execution:
> harness (being it Junit, TestNG or my own script) must understand "word"
> mark-up instead of just simple walking directories.

BTW, after brief browsing of the JUnit 4.0 documentation I didn't find
any extra *grouping* capabilities (comparing to JUnit 3.8.1). Only old
TestSuites. Does anybody have more information about this?

...
> 3. If test characteristics changed we should fix test file, instead of just
> moving file, that is easier to track then changes.

I don't think test characteristics (do you mean groups?) can be
changed by itself. I mean Linux -specific test can't immediately
become Windows-specific, it needs someone to update the code. This
someone can also update the metadata that comes along with the code,
it is easy. The only exception I can think about is when some test
suddenly starts failing. But in this case it is also a good idea to go
and look at the source code of failing tests. And again this will be
the place there the metadata lives – it is easy so change it at once.


> Other pros/cons?

4.  If someone needs the number of failing tests or platform-specific
tests he will need to run the test harness to collect this information
since the metadata is scattered about the source tree.

> I think using metadata in tests vs. physical test location wins.
>  thanks, Vladimir

Well, currently I think the metadata approach better suits the needs
of our project.

Regards,




2006/7/24, Vladimir Ivanov <ivavladimir@gmail.com>:
> Let me add a couple of words.
>
> The problem: in Harmony unit test suite we need to mark-up somehow and
> manage:
> -  Platform-specific unit tests
> -  Harmony implementation test vs. API tests
> -  Broken buggy tests
> -  Tests failing due to bug in implementation
> -  [Potentially in future] type of test – unit, functional, stress,
> reliability, etc.
> -  [Potentially in future] regression test
> -  Each of the listed above can be:
>   - OS-specific
>   - Processor architecture-specific
>
> Metadata vs. directory layout pros:
> 1. Directory structure simplification: we have all tests for the given class
> in one directory, there is no need to have numerous Win/Linux/ia32/ia64/etc.
> subdirectories.
> 2. Lossless test exclusion – we do not exclude the whole TestCase, just
> small tests.
> 3. If we decide to keep all excluded tests in a separate TestCase  *Fail*
> Test.java classes, then, having metadata, there is no need in moving tests
> from one file to another when we want to un-exclude tests (this is risky
> operation which can lead to loosing tests).
> 4. Easy to collect statistics – how many / which tests are: "broken buggy
> tests", Harmony specific, Win-specific, etc.
> 5. Easy to do massive changes if necessary, while no need to re-name
> Win/Linux/ia32/etc. directories or file or move files.
> 6. History of all test changes are stored along with the test file.
>
> Cons:
> 1. More complex algorithms of grouping / selecting tests for execution:
> harness (being it Junit, TestNG or my own script) must understand "word"
> mark-up instead of just simple walking directories.
> 2. People should learn allowed mark-up (list of groups) instead of just
> intuitively put tests into specific directory or TestCase.
> 3. If test characteristics changed we should fix test file, instead of just
> moving file, that is easier to track then changes.
>
> Other pros/cons?
> I think using metadata in tests vs. physical test location wins.
>  thanks, Vladimir
>
> On 7/7/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> >
> > May I tactfully suggest that we get this back to a discussion of the
> > pros and cons of JUnit test suites and/or TestNG metadata vs. directory
> > layout.
> >
> > It sounds like we all want to resolve that problem asap.
> >
> > Regards,
> > Tim
> >
> > George Harley wrote:
> > > Mark Hindess wrote:
> > >> On 6 July 2006 at 18:05, George Harley
> > >> <george.c.harley@googlemail.com> wrote:
> > >>
> > >>> Mark Hindess wrote:
> > >>>
> > >>>> On 6 July 2006 at 12:55, George Harley <
> > george.c.harley@googlemail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Mark,
> > >>>>>
> > >>>>> From what I can tell this JIRA hasn't really achieved much
apart
> > >>>>> from pushing code around the repository and breaking at least
one
> > >>>>> patch (HARMONY-755).
> > >>>>>
> > >>>> Well, obviously that wasn't my motivation! ;-)
> > >>>>
> > >>>>
> > >>> Hi Mark,
> > >>>
> > >>> No one was saying it was. BTW, good to hear you have some motivation
> > :-)
> > >>>
> > >>>
> > >>>
> > >>>> >From the description, it was clear (to me anyway) that the
patch
> > >>>> was to
> > >>>> enable the use of platform-specific test code.  While the directories
> > >>>> for the platform-specific test code are currently empty, I'm certain
> > >>>> Paulex plans to rectify this pretty soon.
> > >>>>
> > >>>>
> > >>> Creating empty directories is not the issue here. The patch also
> > >>> entailed moving a whole bunch of other files around the source tree
> > >>> for reasons that are  currently being discussed in the dev list.
> > >>>
> > >>>
> > >>>> I think Paulex was correct to separate the process of allowing
for
> > >>>> platform-specific tests (HARMONY-782) from any JIRA containing
new
> > >>>> tests.
> > >>>>
> > >>> The "process" of allowing for new platform-specific tests is
> > >>> precisely what is being currently discussed on the dev-list in the
> > >>> referenced thread.
> > >>>
> > >>
> > >> I thought it was categorisation of tests in general.
> > >>
> > >>
> > >
> > > Hi Mark,
> > >
> > > Since "platform-specific" is one important category of test then
> > > discussion and agreement on the general topic is important.
> > >
> > >
> > >>>> The JIRA comment by Paulex mentioned that it would break two existing
> > >>>> JIRA issues - HARMONY-775 and HARMONY-767.  I applied the former
but
> > >>>> the
> > >>>> latter was already assigned to Tim and marked 'In Progress' so
I
> > didn't
> > >>>> feel it was right to steal it.  However I have made the trivial
> > change
> > >>>> to the patch metadata to fix the HARMONY-767 patch.
> > >>>>
> > >>>> Unfortunately it didn't mention the HARMONY-775 patch, otherwise
I
> > >>>> might
> > >>>> have checked with you first.
> > >>>>
> > >>> It was HARMONY-755. I know, now I'm just being picky :-)
> > >>>
> > >>
> > >> Yes. :-)
> > >>
> > >>
> > >>>>> It would be great if you or Paulex (and everyone in fact) could
> > >>>>> comment in the "[classlib] Testing conventions - a proposal"
thread
> > >>>>> [1] about this.
> > >>>>>
> > >>>> Certainly - though this seems to me to be orthogonal to the purpose
> > of
> > >>>> the HARMONY-782 patch.
> > >>>>
> > >>> The summary of HARMONY-782 is "Relayout NIO test cases to platform
> > >>> dependent". That is orthogonal to the dev-list discussion on proposed
> > >>> test layout ??? Are you serious ??????
> > >>>
> > >>
> > >> Ok so maybe not orthogonal but the JIRA (regardless of the exact title)
> > >> was an enabler to allow additional platform-specific tests to be added.
> > >> And adding new tests is something that is independent of the need to
> > >> restructure.  Or are you saying we shouldn't create any more tests (or
> > >> fix existing tests) until the restructuring issue is decided?
> > >>
> > >>
> > >
> > > If adding new platform-specific tests is "independent of the need to
> > > restructure" then why did you restructure the NIO tests ?
> > >
> > >
> > > No, I am not saying that we shouldn't create any more tests. No, I am
> > > not saying that we should stop fixing existing ones. This is not a
> > > restructuring issue. If anything, this is an anti-restructuring issue.
> > > This is about pausing to consider a different approach to the existing
> > > proposal for how we manage our tests. It deserves to be considered as it
> > > has the potential to save us all a lot of time and effort pushing files
> > > around.
> > >
> > >> While I see the importance of the restructuring I'm also keen not to
> > >> prevent the problematic nio tests to be fixed.
> > >>
> > >>
> > >
> > > Ditto. But what is the urgency here ?
> > >> Are you suggesting that applying the JIRA made the state of the tests
> > >> any worse than it was before?  (I even made an effort to ensure that
> > the
> > >> change was made in a way that was more consistent with the current
> > state
> > >> of another module - to make it easier to programmatically fix them
> > later
> > >> when the test structure issue is resolved.)
> > >>
> > >> Regards,
> > >>  Mark.
> > >>
> > >>
> > >
> > > IMHO this is not really about just HARMONY 782 and I would be genuinely
> > > upset if the impression was that I was getting at you or Paulex because
> > > it's not true. This is about asking you, Paulex and everyone to think
> > > about what our tests structure is going to look like eventually, how
> > > much effort is going to be required to maintain its labyrinth layout,
> > > the amount of overhead that is going to mean for our infrastructure (Ant
> > > scripts, IDE metadata files etc) and whether or not we can do better.
> > >
> > >
> > > Best regards,
> > > George
> > >
> > >>
> > >> ---------------------------------------------------------------------
> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >>
> > >>
> > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> > --
> >
> > Tim Ellison (t.p.ellison@gmail.com)
> > IBM Java technology centre, UK.
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>


-- 
Alexei Zakharov,
Intel Middleware Product Division

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message