axis-java-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran" <stev...@iseran.com>
Subject Re: AXIS Test Restructuring
Date Tue, 28 May 2002 20:32:05 GMT
hey, this is a good start. I will look at it in more detail in a day or so,
as usual remind me if you hear nothing.

-steve

----- Original Message -----
From: "Matt Seibert" <mseibert@us.ibm.com>
To: <axis-dev@xml.apache.org>
Sent: Tuesday, May 28, 2002 7:05 AM
Subject: Re: AXIS Test Restructuring


> Steve,
>
> Here is the "high-level" design of my test / samples restructuring effort.
> I began this undertaking with the following input as requirements:
>
>    1. Determine the needs and dependancies of each test and sample
>    2. Reduce the complexity of the xml that compiles, deploys, execs, and
>       undeploys samples and tests
>    3. Enable a new test or sample to be "plugged in" to the suites,
without
>       disturbing currently working things
>    4. Enable the ability to instantiate any single sample or any single
>       test
>    5. Identify tests and samples that need "network" or "internet"
>       resources
>    6. Enable the ability to group tests and samples together in "bundles"
>
> This was based on the following information which was relayed to me:
>    1. Developers don't like to run ALL of the tests, if they are changing
>    only one part
>    2. Developers don't like adding new tests / samples because their
>    integration into the big master build.xml recursive functions
>       threatens to break everything
>    3. Some development organizations test in an "off-line" mode, and
>    therefore cannot exec ALL of the tests ALL of the time.

All good points


> To these ends, I have done the following:
>    1. Broken out the buildTest and samples targets from build.xml into
>    buildTest.xml and buildSamples.xml
>    2. Componentized those big recursive "build everything" calls into
>    specific test compilation xml files
>             For example, the commands / code to build the sessions test is
>    now located in a file, buildComponent.xml, in the directory
>             test/sessions under the target "compile"


>    3. Instead, in the buildTest and buildSamples files, there now exists a
>    non-complex variant of the ant ForeachTask, which finds all files
>       named buildComponent.xml in test/ and samples/ respectively, and
>    operates on them.

ok, an instance of the hypothetical <anton> task.

>             This allows a new test / sample to be "plugged in" to the
test/
>    and samples/ trees, and then be exectued upon, withouth
>             having to muck with the master controlling files

nice.

>    4. If a single test or sample is needed to be instantiated directly (is
>    a pre-req for another test or sample for example), then the
>       developer simply adds a block as below into either buildTest.xml or
>    buildSample.xml to enable the single component use of that
>       specific test or sample
>             <target name="session">
>                   <ant inheritAll="true"
>    antfile="test/session/buildComponent.xml" target="compile"/>
>             </target>

I dont like this style of running a build in a different base directory from
where the file lives, it causes confusion. There are some changes I need to
make to axis' build.xml to make the <ant dir= /> tactic work.

>    5. I am still working on this part. The basic jist is that if there is
>    no internet (network failure, firewalled development environment, etc)
>       the tests should not fail.  I have created a property in the test.
>    properties file which is test.runtest and is given a value of "LOCAL",
>       "REMOTE", or "BOTH" to control this function.  This is one of the
>    parts that I will need some help with.

regarding online/offline, here is the test from ant1.5'd build file

  <!-- test to see if we are online or not. can take a while when we are off
line, so
    setting the property is a good shortcut-->
  <target name="probe-offline">
      <condition property="offline">
      <or>
        <isset property="offline"/>
        <not>
          <http  url="http://www.apache.org/"/>
        </not>
      </or>
    </condition>
    <echo level="verbose" > offline=${offline}</echo>
  </target>

if you run it with -Doffline=true then it is manually off-line, otherwise it
probes for apache and keeps going from there...this property is used to tune
some of the tests:

  <patternset id="onlinetests">
    <exclude name="**/GetTest.java" if="offline" />
  </patternset>

This only works on ant1.5, as only it has the <http> condition, but it works
well. Personally I think any reworked Axis test framework should only use
ant1.5. as its basis; gump uses it after all


>    6. As in item 4, in order to "group" a bunch of tests, this can be
>    implemented in two methods:
>             A. Create a target task in the build<Test,Samples> file that
>    instantiates them all
>             B. Create a second ForeachTask loop that looks for files
called
>    testBundle*.xml and exec each of these.
>
> As for your issue with "stopping on the first failure",  I beleive that
> there is already a flag set up to ignoreFailures in the xml files, which
> will then exec all tests, and give you a wrap up of what did, or did not,
> work.  Also, to that end, I will need to add a "common output" method for
> these componentized tests to allow them (if the user wishes) to have their
> output placed in either separate log file (reports/test-session.xml), a
> "wrap-up" log file (reports/test.xml), or both.  Suggestions and input
here
> would be greatly appreciated.

We have just started discussing on ant-dev where we could go with a
'buildunit' framework; a way of running ant over a set of targets with the
output fed back into junit style XML report docs afterwards, that can be
<style>d into something reasonable.

The other trick would be to take your foreach task, add a failonerror attr,
and have it call ant onto the subbuilds with a new logger to log everything.
Or run ant from the toplevel with its xmllogger, and add a very fancy XSLT
to make something good from the results.

-steve





>
> Also, the buildComponent.xml are all a "template" type files, all with the
> same targets, the same dependancies, and the same ordering, all reading in
> common properties.  In this way it should be much easier to trace through
> all of these tests, simply because they are a small bit easier to
> understand.
>
> Please, everyone feel free to comment openly on this.  Tell me what you
> hate, and what you like, and where you need more information.
>
> Matt Seibert                                           mseibert@us.ibm.com
> IBM        External:    (512) 838-3656      Internal:   678-3656
>
>
> > Prompted by a note from Steve Loughran this morning, and being budged by
> > some of my teammates, I am putting in that I am willing to chip in on
the
> > restructuring.
> >
> > Steve,
> >
> > I have been working for about two weeks on the re-structuring of the
> test/
> > and samples/ compilation / execution process, being able to componentize
> > and instantiate any given test singularly, and making the system easier
> to
> > maintain and, hopefully, a little better organized.
> >
> > I would like to speak with you about how I foresee the build/execution
of
> > the test and samples structures changing in correlation to how you think
> > the overall build process should be changing.  I would very much like
for
> > you to review my proposed changes, as well as see what your proposals
> are,
> > so as to produce the best possible output.
> >
> > Please feel free to ping me back.  I am new to the axis community, and
am
> > not quite sure how we all go about sharing information......
>
> I am new to axis too, but worked on other apache projects. we can discuss
> stuff on the dev mail list as it stores the discussion and lets other
> people
> get involved; it "eliminates surprises"
>
> So, write up your proposed changes and see where you think we should go.
>
> Me, I think that as well as cleaning up the build process, we need to work
> on the test suite so it runs more like a big fat junit test; you run all
> the
> tests and get back an XML file of what worked, what didnt...to date the
way
> we go right now the first failure seems to stops the run.
>
> I dont know quite how to do that (yet), though I have some definate ideas,
> which revolve around how ant's unit tests run build files inside junit,
but
> with a bit more template driven generation of build files, and maybe even
> the junit test cases to call them.
>
> -steve


Mime
View raw message