harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ivanov" <ivavladi...@gmail.com>
Subject Re: [testing] Peace (was: Re: svn commit: r419522 - in /incubator/harmony/enhanced/classlib/trunk/modules/nio: .classpath build.xml make/hyproperties.xml src/test/java/common/ src/test/java/common/org/ src/test/java/linux/ src/test/java/org/ src/test
Date Mon, 24 Jul 2006 13:54:40 GMT
Let me add a couple of words.

The problem: in Harmony unit test suite we need to mark-up somehow and
manage:
-  Platform-specific unit tests
-  Harmony implementation test vs. API tests
-  Broken buggy tests
-  Tests failing due to bug in implementation
-  [Potentially in future] type of test – unit, functional, stress,
reliability, etc.
-  [Potentially in future] regression test
-  Each of the listed above can be:
   - OS-specific
   - Processor architecture-specific

Metadata vs. directory layout pros:
1. Directory structure simplification: we have all tests for the given class
in one directory, there is no need to have numerous Win/Linux/ia32/ia64/etc.
subdirectories.
2. Lossless test exclusion – we do not exclude the whole TestCase, just
small tests.
3. If we decide to keep all excluded tests in a separate TestCase  *Fail*
Test.java classes, then, having metadata, there is no need in moving tests
from one file to another when we want to un-exclude tests (this is risky
operation which can lead to loosing tests).
4. Easy to collect statistics – how many / which tests are: "broken buggy
tests", Harmony specific, Win-specific, etc.
5. Easy to do massive changes if necessary, while no need to re-name
Win/Linux/ia32/etc. directories or file or move files.
6. History of all test changes are stored along with the test file.

Cons:
1. More complex algorithms of grouping / selecting tests for execution:
harness (being it Junit, TestNG or my own script) must understand "word"
mark-up instead of just simple walking directories.
2. People should learn allowed mark-up (list of groups) instead of just
intuitively put tests into specific directory or TestCase.
3. If test characteristics changed we should fix test file, instead of just
moving file, that is easier to track then changes.

Other pros/cons?
I think using metadata in tests vs. physical test location wins.
 thanks, Vladimir

On 7/7/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
>
> May I tactfully suggest that we get this back to a discussion of the
> pros and cons of JUnit test suites and/or TestNG metadata vs. directory
> layout.
>
> It sounds like we all want to resolve that problem asap.
>
> Regards,
> Tim
>
> George Harley wrote:
> > Mark Hindess wrote:
> >> On 6 July 2006 at 18:05, George Harley
> >> <george.c.harley@googlemail.com> wrote:
> >>
> >>> Mark Hindess wrote:
> >>>
> >>>> On 6 July 2006 at 12:55, George Harley <
> george.c.harley@googlemail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Mark,
> >>>>>
> >>>>> From what I can tell this JIRA hasn't really achieved much apart
> >>>>> from pushing code around the repository and breaking at least one
> >>>>> patch (HARMONY-755).
> >>>>>
> >>>> Well, obviously that wasn't my motivation! ;-)
> >>>>
> >>>>
> >>> Hi Mark,
> >>>
> >>> No one was saying it was. BTW, good to hear you have some motivation
> :-)
> >>>
> >>>
> >>>
> >>>> >From the description, it was clear (to me anyway) that the patch
> >>>> was to
> >>>> enable the use of platform-specific test code.  While the directories
> >>>> for the platform-specific test code are currently empty, I'm certain
> >>>> Paulex plans to rectify this pretty soon.
> >>>>
> >>>>
> >>> Creating empty directories is not the issue here. The patch also
> >>> entailed moving a whole bunch of other files around the source tree
> >>> for reasons that are  currently being discussed in the dev list.
> >>>
> >>>
> >>>> I think Paulex was correct to separate the process of allowing for
> >>>> platform-specific tests (HARMONY-782) from any JIRA containing new
> >>>> tests.
> >>>>
> >>> The "process" of allowing for new platform-specific tests is
> >>> precisely what is being currently discussed on the dev-list in the
> >>> referenced thread.
> >>>
> >>
> >> I thought it was categorisation of tests in general.
> >>
> >>
> >
> > Hi Mark,
> >
> > Since "platform-specific" is one important category of test then
> > discussion and agreement on the general topic is important.
> >
> >
> >>>> The JIRA comment by Paulex mentioned that it would break two existing
> >>>> JIRA issues - HARMONY-775 and HARMONY-767.  I applied the former but
> >>>> the
> >>>> latter was already assigned to Tim and marked 'In Progress' so I
> didn't
> >>>> feel it was right to steal it.  However I have made the trivial
> change
> >>>> to the patch metadata to fix the HARMONY-767 patch.
> >>>>
> >>>> Unfortunately it didn't mention the HARMONY-775 patch, otherwise I
> >>>> might
> >>>> have checked with you first.
> >>>>
> >>> It was HARMONY-755. I know, now I'm just being picky :-)
> >>>
> >>
> >> Yes. :-)
> >>
> >>
> >>>>> It would be great if you or Paulex (and everyone in fact) could
> >>>>> comment in the "[classlib] Testing conventions - a proposal" thread
> >>>>> [1] about this.
> >>>>>
> >>>> Certainly - though this seems to me to be orthogonal to the purpose
> of
> >>>> the HARMONY-782 patch.
> >>>>
> >>> The summary of HARMONY-782 is "Relayout NIO test cases to platform
> >>> dependent". That is orthogonal to the dev-list discussion on proposed
> >>> test layout ??? Are you serious ??????
> >>>
> >>
> >> Ok so maybe not orthogonal but the JIRA (regardless of the exact title)
> >> was an enabler to allow additional platform-specific tests to be added.
> >> And adding new tests is something that is independent of the need to
> >> restructure.  Or are you saying we shouldn't create any more tests (or
> >> fix existing tests) until the restructuring issue is decided?
> >>
> >>
> >
> > If adding new platform-specific tests is "independent of the need to
> > restructure" then why did you restructure the NIO tests ?
> >
> >
> > No, I am not saying that we shouldn't create any more tests. No, I am
> > not saying that we should stop fixing existing ones. This is not a
> > restructuring issue. If anything, this is an anti-restructuring issue.
> > This is about pausing to consider a different approach to the existing
> > proposal for how we manage our tests. It deserves to be considered as it
> > has the potential to save us all a lot of time and effort pushing files
> > around.
> >
> >> While I see the importance of the restructuring I'm also keen not to
> >> prevent the problematic nio tests to be fixed.
> >>
> >>
> >
> > Ditto. But what is the urgency here ?
> >> Are you suggesting that applying the JIRA made the state of the tests
> >> any worse than it was before?  (I even made an effort to ensure that
> the
> >> change was made in a way that was more consistent with the current
> state
> >> of another module - to make it easier to programmatically fix them
> later
> >> when the test structure issue is resolved.)
> >>
> >> Regards,
> >>  Mark.
> >>
> >>
> >
> > IMHO this is not really about just HARMONY 782 and I would be genuinely
> > upset if the impression was that I was getting at you or Paulex because
> > it's not true. This is about asking you, Paulex and everyone to think
> > about what our tests structure is going to look like eventually, how
> > much effort is going to be required to maintain its labyrinth layout,
> > the amount of overhead that is going to mean for our infrastructure (Ant
> > scripts, IDE metadata files etc) and whether or not we can do better.
> >
> >
> > Best regards,
> > George
> >
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message