axis-java-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject cvs commit: xml-axis/java/docs/docbook testing-again.dbk
Date Thu, 19 Jun 2003 05:12:11 GMT
stevel      2003/06/18 22:12:11

  Added:       java/docs/docbook testing-again.dbk
  First docbook addition to the repository, goal would be to migrate all the docs from HTML
to .dbk, though not with the app I used to write this. Maybe OpenOffice with DBK support is
  The subject of the doc is of course some thoughts on testing. I dont come to any compelling
answers, but do try and scope out all the ways we really need to test Axis and suggest some
changes to what we have today. It should be sufficient to start discussion :)
  Revision  Changes    Path
  1.1                  xml-axis/java/docs/docbook/testing-again.dbk
  Index: testing-again.dbk
  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
    <title>Testing (again)</title>
      <para><author><lineage>Steve Loughran</lineage></author></para>
      <title>Where we are today?</title>
      <para>We are now at the second revision of the Axis test framework. The
      original design had all classes in a package under java/test, built them
      in one big &lt;javac&gt; and then executed them. While it ran the tests,
      it was not that flexible. It was hard to run individual tests, and it
      was hard to maintain. A major effort by Matt Siebert refactored the test
      process to address these.</para>
      <para>The revision attempted to address this with a modular design, based
      on separate Ant build files for each test package, using common XML entity
      references to share build file fragments between the files. This gave us
      isolated builds of each subcomponents, the ability to build and run tests
      on their own, and the flexiblity to have very different build processes
      for each of the tests.</para>
      <para>The many build files compile the source, all of java/test/*.java
      into build/classes, putting them into the hierarchy build/classes/test.
      Test packages which have special dependencies can make their builds
      conditional, so only those tests for which we have support get compiled
      down. This architecture makes it easy to add a new test package, without
      having to edit shared build files.</para>
      <para>We have a separation between "unit tests" and
      "functional" tests; the latter includes the interop and attachment
      tests. There are separate targets in build test to build them, as the
      choice of which tests to execute is primarily driven by the compilation
      process. Nearly all the tests in the class tree will get executed, so to
      select which tests to run, you control which tests get built. This is a
      simple way of letting the test package-specific build files control which
      test to run.</para>
        <title>The Tests</title>
        <para>The many build files compile the source, all of java/test/*.java
        into build/classes, putting them into the hierarchy build/classes/test.
        Test packages which have special dependencies can make their builds
        conditional, so only those tests for which we have support get compiled
        <para>We have a separation between "unit tests" and
        "functional" tests; the latter includes the interop and
        attachment tests. There are separate targets in build test to build
        them, as the choice of which tests to execute is primarily driven by the
        compilation process. Nearly all the tests in the class tree will get
        executed, so to select which tests to run, you control which tests get
        built. This is a simple way of letting the test package-specific build
        files control which test to run.</para>
        <para>A core component of many of the tests is generating Java source
        from WSDL files, both local test case WSDL and remote interop WSDL. The
        latter introduces a requirement to be on line, and on-line through a
        transparent firewall -we dont look after proxy settings enough to run
        behind a firewall whose sole net access is via a proxy 80. This is
        somewhat ironic, given that such a facility is the selling point of the
        transport-stack-atop-port-80 that is the SOAP and WS-* specification
        <para>As well as lacking off-line support for generating WSDL, we cant
        (obviously) run the interop tests without a network connection. This
        means that when a remote interop server goes down, the build fails.</para>
        <para>After compiling all the code down, we run the tests. This is done
        by batch JUnit execution of all the test suites in all the packages with
        a PackageTests.class class in their package (i.e. all of
        <para>Functional tests are all of **/FunctionalTests.class and
        **/*TestCase.class; the latter are those test cases which are
        auto-created by the Wsdl2Java routine, often with manual editing to make
        the test complete.</para>
        <para>When the tests need a functional servlet engine to host the web
        services, we bring up the simple axis server; a minimal implementation
        of the servlet API that omits the production-quality aspects of a web
        server, including JSP support. The &lt;runaxisfunctionaltests&gt; task
        starts and stops the server, using an execution process borrowed from
        Cactus: we supply the task with the target names of the start and stop
        targets, and the task executes them before and after running all the
        functional tests.</para>
        <title>Result Processing</title>
        <para>In a Gump build, the build stops after the first failure, and the
        team notified. The property <varname></varname> sets
        the <symbol>haltonfailure</symbol> attribute of the &lt;junit&gt;
        task; set it to true and the test suite runs all tests before
        completing. Either way, the <symbol>create-test-report</symbol> target
        will, if Xalan or other XSLT engine is present, convert the XML reports
        of the test run into an HTML report, package by package.</para>
      <title>What do we want from a test suite?</title>
        <title>Basic Improvements to the current status quo</title>
            <para>All the tests to pass :)</para>
            <para>Faster tests</para>
            <para>Scalability: easy to add new tests</para>
            <para>Offline support, and robustness against unavailable interop
        <title>Functional testing on production app servers</title>
        <para>If we look at a lot of bugreps they are related to config and
        operations on app servers. "Weblogic doesnt save
        server-config.wsdd", "SunOne server has the wrong
        classpath", " won't compile .JWS pages that import
        javax.servlet.*", etc, etc. We need to run more tests on production
        systems, rather than wait for user feedback after we ship. Now everybody
        runs their apps on some such systems, so we have implicit testing, but
        it is not part of the daily Gump or any other regular, controlled test
        <para>We could modify the test system so that instead of starting the
        SimpleAxisServer servlet routine, we can deploy to a local web server or
        app server. This would verify that the core test suite runs on different
        <title>Test more than SOAP</title>
        <para>We need more tests to validate the configuration; extending the
        httpunit tests to have more tests of not-quite-SOAP requests. What
        happens when the server gets less than they were promised? What about
        more than promised? near-infinite recursive XML? xsd:import statements
        in the XML? What happens when a client starts parsing from a socket that
        doesnt close its connection, or lies about the amount it is sending?</para>
        <para>These are the security and robustness categories we aren't
        testing for today.</para>
        <title>Automated invocation of compliance test suites: JAX-RPC TCK, WS-I
        <para>We have one test suite, JAX-RPC, that is only available under
        restricted conditions. We need someone with access to the test suite to
        run it.</para>
        <title>Understandig that Interop servers are regularly unavailable</title>
        <para>If Axis depends on everyone's interop server to be present,
        then we have a global build system that breaks every time somebody in
        their machine off -"the light switch in belgium problem". This
        is too brittle. We need to cache the external WSDL in CVS, then probe
        the servers to see if that has changed, downloading it only if it is
        different. It would be nice to only regenerate the java proxy classes
        from the WSDL when such a change has occurred.</para>
        <title>Load testing</title>
        <para>What happens to the system under load? This is very dependent upon
        the app server; having tests running on the production server is a first
        step to this. Traditional load testing has N clients each making M
        requests, for N*M simultaneous requests. The facilities for individuals
        to perform aggressive load tests are limited, but there is strength in
        numbers; many Axis developers could have their test systems sychronised
        to test an externally visible server together. This co-ordination could
        be though a P2P infrastructure, such as JXTA or Chord, but as we are not
        trying to design a stealth DDoS system, we could do it client-server
        with a (separate) SOAP service choreographing the clients.</para>
        <para>This would seem to be a long term project.</para>
        <title>Performance testing</title>
        <para>This is related to load testing, but doesnt need as many clients.
        We need to know how long it takes for actions to be executed, and we
        need to log this over time so that major regressions can get picked up.
        If on the daily run one test takes 50% longer than usual, it is
        immediately obvious that one of the day's changes caused the
        problem. If the performance of the system doesn't get looked at till the
        next version goes into RC phase, performance slippages get missed, and
        even institutionalised.</para>
        <title>Coverage Analysis</title>
        <para>We should be able to use quilt
        (<ulink url=""></ulink>)
        to do this today. As with performance tests, tracking coverage changes
        over time is informative; it shows whether things are improving or
        getting worse.</para>
        <title>Local interop testing with non-Axis clients and servers</title>
        <para>We already have some examples of .net client tests against axis,
        with an Ant build but sadly no automatic .NUnit test case generation. We
        can also build axis clients against .Net and other servers, where we can
        create JUnit stubs for ease of test generation. If such tests were part
        of the main test suite, then suitably equipped systems could run our own
        interop tests. These would be an extension of the SoapBuilders main
        suite. Here we"d want to verify that fixes worked and continued to
        work (e.g. the .NET1.0 empty array bugfix). We can also add stuff that
        isn't in the SoapBuilders: Cookie Sessions, Http Header sessions,
        is-date-in-the-right-TZ-at-the-far-end tests, and so on.</para>
        <para>There are logistical/tactical and strategic arguments against
        doing this. Logistical: even for the example of one client platform is
        daunting; we don't want to expose a .NET server to the internet for
        anyone to hit it, so the tests will only run on localhost when
        localhost=windows, which excludes the Gump builds.</para>
        <para>The strategic argument is that the combinatorial explosion of
        local interop testing against multiple clients and servers is too big;
        that is what the SoapBuilders are for. Either we focus on one or two key
        platforms to interop test against and MSSTK, or we raise the
        problem back to SoapBuilders.</para>
        <para>What would we want from SoapBuilders, to help our regression and
        interop problems? I'd argue for extra tests, above and beyond the
        formal "rounds", wherever someone has an interop issue. We
        should be able to announce that we have a problem and the URL of a test
        endpoint, and everyone can add it to the things we can test against.
        Similarly, other platforms should not just fix things, they should
        provide means for outsiders to test the system.</para>
        <para>Glen Daniels has proposed a pattern-matching server that waits for
        expected wire traces, and generates preprogrammed responses, simulating
        part of the behaviour of a foreign server. This has the advantage of
        being standalone, but with the disadvantage of not being as thorough as
        a live test. You also have the challenge of keeping the datasets up to
        <title>Wiretrace logging in the test case results</title>
        <para>This is just an extra little feature for diagnosis: can we record
        the wire traces of sent and received messages, and whenever we get a
        test failure, save the results to the JUnit log for an easier
        post-mortem. Just a thought :)</para>
        <title>Ease of learning, installation, use</title>
        <para>We are an open source project where anyone can download the
        source, build it and run the tests. Therefore the test framework must be
        easy to run, easy to work with, and easy to maintain by people other
        than the original authors. We also want to keep effort minimised by
        re-using as much as possible of other OSS projects.</para>
      <para>Here are some of the things we can do</para>
        <para>Leave things as they are. Maybe move to Ant1.6 alpha builds to get
        better memory management, the faster build.xml parser and the refactored
        .NET tasks, or just up to 1.5.3/1.5.4 to get the memory leak fix.</para>
        <para>Costs: nothing at first; a gradual increase in longer term costs.</para>
        <title>Improve build.xml reuse</title>
        <para>We don't necessarily need a separate build file in every test
        package. Instead we can have a properties file in there that sets well
        known properties</para>
        <para>This can be read in by something (ant or custom java) and used to
        control the build. Reading it into pure ant (i.e. without writing our
        own task) would be tricky as condition expressions are tricky. An XML
        description might be better, and could be XSLT'd into the build
        <title>Caching WSDL Generation</title>
        <para>This is a trick we can do with any of these.</para>
        <para>Write a new &lt;axis-importwsdl&gt; task that implements
        dependency awareness around the inner generation process. We could go
        one step further and integrate dependency logic into the generation
        process, but as that is more fundamental, I am a bit leery of that.
            <para>caches the results of the fetch.</para>
            <para>uses the if-modified-since tag to conditionally retrieve
            <para>even if content is returned, compares it byte-for-byte against
            the cached copy</para>
            <para>only imports the wsdl if the wsdl file is newer than a
            timestamp marker file in the generated dir (and a force option is
            <para>if the server is unreachable, but the cached copy is present,
            don't fail the build, just set a property saying the server isn't
            there and continue with the WSDL generation.</para>
            <para>if the server is unreachable and the cached copy isn't there,
            only fail the build if some attribute is set, otherwise a
            wsdlnotpresent property is set</para>
            <para>We could do this over a fairly convoluted set of ant 1.6
            tasks, but only if the httptasks in the Ant sandbox were pulled in for
            more graceful handling of missing servers. Pulling it in to one Axis
            task gives us more control, ant1.5 support and wouldn't be that
            hard to implement.</para>
        <title>Write our own test harness</title>
        <para>This deserves a mention: could we do our own complete test
        harness? Why? is the reponse. What would that add?</para>
        <para>In theory, having our own hosting app would let us run tests
        differently from core JUnit, doing more SOAP related things like post
        XML and expect responses.</para>
        <title>Return to being JUnit-centric</title>
        <para>The advantage of an ant-centric test system is that we can use Ant
        to build stuff during testing. The disadvantage is in complexity and
        time in running the tests. Is there a better compromise? Maybe. </para>
        <para>It is possible to run Ant from inside JUnit. This is how Ant runs
        its many self tests; by invoking Ant from JUnit itself. If we
        put JUnit in charge, then those tests that do need a complex Ant-based
        test system can call Ant to aid it, the rest run straight from JUnit.</para>
        <para>We may be able to take advantage of this by categorising tests
        into various patterns that we can build and run differently.</para>
            <para>Pure unit tests that compile and run without needing any
            server at all</para>
            <para>WSDL-importing tests that need to import WSDL and generate
            code before the tests run</para>
            <para>Local functional tests that run against an instance of the
            Axis running on a servlet engine</para>
            <para>Local functional tests that only run on a full J2EE app server</para>
            <para>Remote interop tests</para>
        <para>Clearly this stuff is not 100% exclusive: a lot of the local
        functional tests generate WSDL first, as do all the interop tests. And
        there are other flags which will include/exclude test items: the
        presence/absence of needed libraries, such as attachment support, and an
        online/offline flag to distinguish tests that need a full internet
        connection, from those that don't. All the interop tests are online, but
        so are a few of the others.</para>
        <para>In a JUnit-centric world, first the local unit tests would get
        built and run, all in in a couple of big &lt;javac&gt; and
        &lt;junit&gt; task. Then the WSDL import process can take place, using
        something like the new dependency-aware wsdl import task proposed
        earlier. </para>

View raw message