axis-java-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Seibert" <>
Subject Re: AXIS Test Restructuring
Date Tue, 28 May 2002 14:05:29 GMT

Here is the "high-level" design of my test / samples restructuring effort.
I began this undertaking with the following input as requirements:

   1. Determine the needs and dependancies of each test and sample
   2. Reduce the complexity of the xml that compiles, deploys, execs, and
      undeploys samples and tests
   3. Enable a new test or sample to be "plugged in" to the suites, without
      disturbing currently working things
   4. Enable the ability to instantiate any single sample or any single
   5. Identify tests and samples that need "network" or "internet"
   6. Enable the ability to group tests and samples together in "bundles"

This was based on the following information which was relayed to me:
   1. Developers don't like to run ALL of the tests, if they are changing
   only one part
   2. Developers don't like adding new tests / samples because their
   integration into the big master build.xml recursive functions
      threatens to break everything
   3. Some development organizations test in an "off-line" mode, and
   therefore cannot exec ALL of the tests ALL of the time.

To these ends, I have done the following:
   1. Broken out the buildTest and samples targets from build.xml into
   buildTest.xml and buildSamples.xml
   2. Componentized those big recursive "build everything" calls into
   specific test compilation xml files
            For example, the commands / code to build the sessions test is
   now located in a file, buildComponent.xml, in the directory
            test/sessions under the target "compile"
   3. Instead, in the buildTest and buildSamples files, there now exists a
   non-complex variant of the ant ForeachTask, which finds all files
      named buildComponent.xml in test/ and samples/ respectively, and
   operates on them.
            This allows a new test / sample to be "plugged in" to the test/
   and samples/ trees, and then be exectued upon, withouth
            having to muck with the master controlling files
   4. If a single test or sample is needed to be instantiated directly (is
   a pre-req for another test or sample for example), then the
      developer simply adds a block as below into either buildTest.xml or
   buildSample.xml to enable the single component use of that
      specific test or sample
            <target name="session">
                  <ant inheritAll="true"
   antfile="test/session/buildComponent.xml" target="compile"/>
   5. I am still working on this part. The basic jist is that if there is
   no internet (network failure, firewalled development environment, etc)
      the tests should not fail.  I have created a property in the test.
   properties file which is test.runtest and is given a value of "LOCAL",
      "REMOTE", or "BOTH" to control this function.  This is one of the
   parts that I will need some help with.
   6. As in item 4, in order to "group" a bunch of tests, this can be
   implemented in two methods:
            A. Create a target task in the build<Test,Samples> file that
   instantiates them all
            B. Create a second ForeachTask loop that looks for files called
   testBundle*.xml and exec each of these.

As for your issue with "stopping on the first failure",  I beleive that
there is already a flag set up to ignoreFailures in the xml files, which
will then exec all tests, and give you a wrap up of what did, or did not,
work.  Also, to that end, I will need to add a "common output" method for
these componentized tests to allow them (if the user wishes) to have their
output placed in either separate log file (reports/test-session.xml), a
"wrap-up" log file (reports/test.xml), or both.  Suggestions and input here
would be greatly appreciated.

Also, the buildComponent.xml are all a "template" type files, all with the
same targets, the same dependancies, and the same ordering, all reading in
common properties.  In this way it should be much easier to trace through
all of these tests, simply because they are a small bit easier to

Please, everyone feel free to comment openly on this.  Tell me what you
hate, and what you like, and where you need more information.

Matt Seibert                                 
IBM        External:    (512) 838-3656      Internal:   678-3656

> Prompted by a note from Steve Loughran this morning, and being budged by
> some of my teammates, I am putting in that I am willing to chip in on the
> restructuring.
> Steve,
> I have been working for about two weeks on the re-structuring of the
> and samples/ compilation / execution process, being able to componentize
> and instantiate any given test singularly, and making the system easier
> maintain and, hopefully, a little better organized.
> I would like to speak with you about how I foresee the build/execution of
> the test and samples structures changing in correlation to how you think
> the overall build process should be changing.  I would very much like for
> you to review my proposed changes, as well as see what your proposals
> so as to produce the best possible output.
> Please feel free to ping me back.  I am new to the axis community, and am
> not quite sure how we all go about sharing information......

I am new to axis too, but worked on other apache projects. we can discuss
stuff on the dev mail list as it stores the discussion and lets other
get involved; it "eliminates surprises"

So, write up your proposed changes and see where you think we should go.

Me, I think that as well as cleaning up the build process, we need to work
on the test suite so it runs more like a big fat junit test; you run all
tests and get back an XML file of what worked, what date the way
we go right now the first failure seems to stops the run.

I dont know quite how to do that (yet), though I have some definate ideas,
which revolve around how ant's unit tests run build files inside junit, but
with a bit more template driven generation of build files, and maybe even
the junit test cases to call them.

View raw message