Mailing-List: contact general-help@xml.apache.org; run by ezmlm
Importance: Normal
Subject: Test Infrastructure Project Proposal
To: "Erich_Gamma@oti.com" <Erich_Gamma@oti.com>,
        "kentbeck@csi.com" <kentbeck@csi.com>
Cc: general@xml.apache.org, general@jakarta.apache.org, pmc@xml.apache.org,
        xerces-j-dev@xml.apache.org, xalan-dev@xml.apache.org,
        cocoon-dev@xml.apache.org
Message-ID: <OF58FB5919.357FB82E-ON852569F1.0076C4D2@raleigh.ibm.com>
From: "Sam Ruby" <rubys@us.ibm.com>
Date: Mon, 12 Feb 2001 16:42:01 -0500
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii

Hi Erich, Kent.

There is a discussion going on in the general@xml.apache.org mailing list
that I would like to invite you two to participate in.

I am copying a large number of mailing lists as the original e-mail
forwarded below has triggered a lot of separate discussions, but I would
like to ask that you follow Scott's request and respond only to
general@xml.apache.org.

- Sam Ruby

---------------------- Forwarded by Sam Ruby/Raleigh/IBM on 02/12/2001
04:37 PM ---------------------------

Scott_Boag@lotus.com on 02/10/2001 04:06:38 PM

Please respond to general@xml.apache.org

To:   general@xml.apache.org, general@jakarta.apache.org
cc:   pmc@xml.apache.org, xerces-j-dev@xml.apache.org,
      xalan-dev@xml.apache.org, cocoon-dev@xml.apache.org
Subject:  Test Infrastructure Project Proposal


Ross Burton <ross.burton@mail.com> wrote:
> Having spent all day upto my arm pits in bits of cocoon, xerces and
> xalan, I've come to the conclusion that tracking down bugs in bits of
> cocoon can be nearly impossible at the moment. How would you guys feel
> about us starting to use JUnit to run unit tests over cocoon?

First, sorry for the large cross-posting.  But I wanted to make sure this
note is received by a large audience.  Replies should be sent only to
general@xml.apache.org (I'm not subscribed to general@jakarta.apache.org).

Xalan being a project that 1) requires a *lot* of testing, and 2) is
sandwiched inbetween Cocoon and Xerces, 3) is dependent on other
technologies like BSF, and 4) is used in several other pipeline scenarios,
we (i.e. the folks at Lotus who are involved in the Xalan project) have
been doing a lot of thinking about this subject.

What is happening in the XML/Web world is the integration of a lot of
smaller components, plugged together via (hopefully) standard interfaces.
This increases the need for unit testing, and integration testing in a big
way.  In systems such as we are building, robustness is everything, and
fragility is becoming an increasing problem.  I feel this is probably the
most critical issue xml.apache.org and the Jakarta projects are facing,
even above performance issues.  The days have ended when Xerces can release
without testing with Xalan, and Xalan can release without testing with
Cocoon, etc.

Also, I feel what we are all practising is pretty close to "Extreme
Programming" (http://www.extremeprogramming.org/), by our very motto of
"release early and often".  Extreme programming is very reliant on having a
*lot* of tests that are constantly run (see
http://www.extremeprogramming.org/rules/unittests.html and
http://www.extremeprogramming.org/rules/functionaltests.html).  This means
that tests must be very fast to create, easy to plug in, easy to have them
become part of the perminant acceptence tests, running the tests must be
extremely convenient, and diagnosing problems from the reports must also be
easy.  And when a bug is found by a user, a test should almost always be
added to the acceptence tests
(http://www.extremeprogramming.org/rules/bugs.html).

We have analyzed JUnit and feel it doesn't address our needs, nor the
integration testing needs, though we like it's Ant base.

I propose a project for testing infrastructure, that covers the needs of
unit testing, stress testing, performance testing, negative testing (i.e.
testing of error conditions), integration testing, and error logging and
reporting.  I don't know or care if this is a Jakarta project or an
xml.apache.org project.  I believe the Jakarta project has been thinking
somewhat along these lines?

The Xalan project already has a fair amount of infrastructure that we would
be happy to contribute.  But basically, I think we should start first with
requirements, then a schema for reporting (i.e. design the data first), go
next to interfaces, and then decide what existing code can be used.

Thoughts?  Does anyone want to -1 this?  If not, where should it live? (I
suspect the answer is in Jakarta, next to Ant).  What should it be named?
What are the next steps?  Who should be the founders?  And what about
C-language integration testing, as well as other languages (which might
argue against a home in Jakarta?)?

Again, sorry for the large cross-posting, but I think it's time for this
issue to get full attention from all the projects.

-scott


                    Ross Burton
                    <ross.burton@        To:     cocoon-dev@xml.apache.org
                    mail.com>            cc:     (bcc: Scott
Boag/CAM/Lotus)
                    Sent by:             Subject:     Re: [C2] Unit
testing.
                    ross@itzinter
                    active.com


                    02/10/2001
                    09:37 AM
                    Please
                    respond to
                    cocoon-dev


Paul Russell wrote:
>
> Guys,
>
> Having spent all day upto my arm pits in bits of cocoon, xerces and
> xalan, I've come to the conclusion that tracking down bugs in bits of
> cocoon can be nearly impossible at the moment. How would you guys feel
> about us starting to use JUnit to run unit tests over cocoon? That way
> we can be a lot more confident that we didn't just break something when
> we make a change? The tests can range from checking that a matcher
> matches, to checking that a classloader class loads, to checking that
> the output of a particular pipeline is as expected. I'm very happy to
> help in this respect, and I have minions (eh, boss?) that will put some
> work into developing tests, too. How do you guys feel about this?
>
> Does anyone know whether the IBM Public Licence is APL compatible, or
> would we have to keep JUnit out of the repository?

A small note - didn't the Avalon list seperate their testing code from
the core and put it on Sourceforge?  arrowhead.sourceforge.net IIRC.

Ah - just went there.  Yes, that is the site of the code (Kevin Burton
is the developer) but there is no downloads, no web page, no nothing.
Anyone know what happened?  Did it get dropped?

Apart from that, +1 for the tests.

Ross Burton

---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@xml.apache.org
For additional commands, e-mail: general-help@xml.apache.org