geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Blevins <david.blev...@visi.com>
Subject Re: Test strategy
Date Mon, 30 Jan 2006 21:09:03 GMT
It's probably going to be you doing this work, so ultimately, you are  
the one who decides what you want to write.  I will just say that you  
and I share the same goals, but your proposal goes against the grain  
of maven which fundamentally combines build/test.  Anytime you do  
that, it's painful.

So let me just stop proposing and do my best to provide the  
information you'd need to get something done.

On Jan 30, 2006, at 11:09 AM, Prasad Kashyap wrote:
> 1. Build and Test will be kept separate functionally.
>
> 2. The itests 'subproject' can scale better. The testbucket could
> become potentially large as more modules start adding their own itests
> or increasing the number of test scenarios.

A point of clarification.  There is no technical limit to  
modularizing and building a set of tests in one area and reusing the  
artifacts produced by them elsewhere.  In other words, where you keep  
the itests does not dictate where you run them.  It is possible to  
build tests in maven 2 and run them in maven 1.  The reverse could be  
true if a maven 2 version of the deployment plugin was created.

> 3. The throughput of the itests will not determine the throughput of
> the G build. Even if we do fix the openejb itests in G now to make it
> very fast, when we have a potentially large number of itests
> contributed by different folks for different modules, they tend to add
> up.

There are logically two different "build times" to consider.  I would  
consider Developer build time to be more precious, and Automated  
built time to be less precious.  Making the running of itests  
optional to developers is mandatory.  Making the itests off by  
default may be a good idea.

Either way, there is no technical limitation to achieving the above  
goals and running itest "on" or "during" the assemblies.

> 4. It will be difficult to selectively include/exclude a few tests.

This is where you pick your poison so to speak.  Not all itest will  
apply to each assembly and vice versa.  Meaning, there are only two  
ways to skin this cat.

   1. You make each assembly say which tests apply to it
   2. You make each set of tests say which assemblies apply to it

With #1 you will have challenges specifying the tests to run.  You  
will have to deploy several apps and run distinct sets of tests.   
With three assemblies and four distinct sets of tests (for example),  
you will unpack and start each server once and deploy into it four  
times (essentially).  A total time of  3 unpacks and startups and 12  
deploys.

With #2 you will have challenges specifying on which assemblies the  
distinct set of tests should run against.  The distinct set of tests  
will have to unpack and start each assembly to run its set of tests.   
With four distinct sets of tests and three assemblies (same example),  
you will unpack, start and deploy on each assembly once for that set  
of tests.  A total of 12 unpacks, starts and deploys will be required.

> This is how I'd imagined it, at a very high level.
> - The itests subproject will have it's own set of modules. These are
> the modules in G that have dropped their itests into this subproject.

Totally possible.

> - The tests will be grouped logically into suites or categories. Each
> individual test will fall in one of these suites.

Not sure of your usage of the word "test" in this sentence.  In junit  
terms is that TestSuite, TestCase, or individual test method.  Can  
you clarify?

> - The tests can all be run or only a subset of suites can be run  
> quite easily.

When you say "easily" do you mean by virtue of them being separate or  
some other idea?

> - Continuum will continue to build G first and then run itests on the
> assemblies.
> - If the assembly passes a certain set of tests, then it can be
> certified good and then be  uploaded to the repository.

The maven 2 lifecycle supports this flow on a per module basis  
(truncated, but still in order):

   compile
   test-compile
   test
   package
   before-integration-test
   integration-test
   after-integration-test
   verify
   install
   deploy

If there is a failure the next phase of the lifecycle is not called.   
So if "test" fails, "package" is not called.  If "integration-test"  
fails then "install" or "deploy" are not called.

    Note, there are no "integration-test-complile" or "integration- 
test-package"
    phases as it is expected these exist in other modules.

The life cycle of assembly SuperServer could work in maven 2 as follows:

Geronimo :: Assembly :: Super Server

   compile        (no op)
   test-compile   (no op)
   test           (no op)
   package        build the assembly
   before-integration-test    unpack and start the server, deploy  
required itest components
   integration-test           run all itests applicable for this  
assembly
   after-integration-test     stop server
   verify         run any additional checks to verify the package is  
valid and meets quality criteria.
   install        put in local maven repo
   deploy         upload to public repo


If you wish to split out integration-test from the maven lifecycle  
and delay publishing the SuperServer binary, you would need at least  
three modules to achieve the same.

Geronimo :: Assembly :: Super Server

   compile        (no op)
   test-compile   (no op)
   test           (no op)
   package        build the assembly
   before-integration-test    (no op)
   integration-test           (no op)
   after-integration-test     (no op)
   verify         (no op)
   install        put in local maven repo
   deploy         WARNING: don't call this our it will be upload to  
public repo without testing

Geronimo :: iTest :: Super Server

   compile        (no op)
   test-compile   (no op)
   test           (no op)
   package        (no op)
   before-integration-test    unpack and start the SuperServer,  
deploy required itest components
   integration-test           run all itests applicable for this  
assembly
   after-integration-test     stop the SuperServer
   verify         run any additional checks to verify the package is  
valid and meets quality criteria.
   install        (no op)
   deploy         (no op)

Geronimo :: Deploy :: Super Server

   compile        (no op)
   test-compile   (no op)
   test           (no op)
   package        (no op)
   before-integration-test    (no op)
   integration-test           (no op)
   after-integration-test     (no op)
   verify         devise some way to verify the itests have run  
successfully on this exact SuperServer binary
   install        (no op)
   deploy         upload to public repo

You will have to do a lot of tricking on maven to do things like  
this, but we've proven that with enough hacking you can push maven  
against it's grain.

> - a download page will be available to download assemblies and
> binaries from every single build. The builds that passed itests will
> be clearly marked so.

Great.


-David

Mime
View raw message