geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasad Kashyap <goyathlay.geron...@gmail.com>
Subject Re: Test strategy
Date Tue, 31 Jan 2006 03:43:13 GMT
David,

Thanks for letting me make that decision. I, at this point, would now
have to get the insight and information from people who are more
knowledgeable about the project; something similiar to what you have
provided now. With my limited knowledge, I don't want to  go down the
wrong path.

I would like to solicit the views of others too like you and get their
views and opinions.

>> - The tests can all be run or only a subset of suites can be run
>> quite easily.

> When you say "easily" do you mean by virtue of them being separate or
> some other idea?

When I said we could run a subset of tests "easily" when the itests
are run "on" assemblies, I meant that it would be relatively easier to
selectively include/exclude tests (by categories) than when compared
to doing the same if the itests are embedded with their modules.

>> - The tests will be grouped logically into suites or categories. Each
>> individual test will fall in one of these suites.

> Not sure of your usage of the word "test" in this sentence.  In junit
> terms is that TestSuite, TestCase, or individual test method.  Can
> you clarify?

I meant TestCase. For. eg, we could have a TestSuite called, "System"
which will have test cases from many different modules. We could say
that these form the core set of tests. If the System suite passed, we
have somewhat of a stable binary with the caveat that all functions
may not work.


In summary, the cons of running the itests "on" assemblies is that it
goes against the m2 lifecycle. And even though integration-test is an
m2 lifecycle phase, we are not exploiting it's usage but calling it
explicity again. The same has to be done for the deploy phase too.

Thanks again
Prasad.

On 1/30/06, David Blevins <david.blevins@visi.com> wrote:
> It's probably going to be you doing this work, so ultimately, you are
> the one who decides what you want to write.  I will just say that you
> and I share the same goals, but your proposal goes against the grain
> of maven which fundamentally combines build/test.  Anytime you do
> that, it's painful.
>
> So let me just stop proposing and do my best to provide the
> information you'd need to get something done.
>
> On Jan 30, 2006, at 11:09 AM, Prasad Kashyap wrote:
> > 1. Build and Test will be kept separate functionally.
> >
> > 2. The itests 'subproject' can scale better. The testbucket could
> > become potentially large as more modules start adding their own itests
> > or increasing the number of test scenarios.
>
> A point of clarification.  There is no technical limit to
> modularizing and building a set of tests in one area and reusing the
> artifacts produced by them elsewhere.  In other words, where you keep
> the itests does not dictate where you run them.  It is possible to
> build tests in maven 2 and run them in maven 1.  The reverse could be
> true if a maven 2 version of the deployment plugin was created.
>
> > 3. The throughput of the itests will not determine the throughput of
> > the G build. Even if we do fix the openejb itests in G now to make it
> > very fast, when we have a potentially large number of itests
> > contributed by different folks for different modules, they tend to add
> > up.
>
> There are logically two different "build times" to consider.  I would
> consider Developer build time to be more precious, and Automated
> built time to be less precious.  Making the running of itests
> optional to developers is mandatory.  Making the itests off by
> default may be a good idea.
>
> Either way, there is no technical limitation to achieving the above
> goals and running itest "on" or "during" the assemblies.
>
> > 4. It will be difficult to selectively include/exclude a few tests.
>
> This is where you pick your poison so to speak.  Not all itest will
> apply to each assembly and vice versa.  Meaning, there are only two
> ways to skin this cat.
>
>    1. You make each assembly say which tests apply to it
>    2. You make each set of tests say which assemblies apply to it
>
> With #1 you will have challenges specifying the tests to run.  You
> will have to deploy several apps and run distinct sets of tests.
> With three assemblies and four distinct sets of tests (for example),
> you will unpack and start each server once and deploy into it four
> times (essentially).  A total time of  3 unpacks and startups and 12
> deploys.
>
> With #2 you will have challenges specifying on which assemblies the
> distinct set of tests should run against.  The distinct set of tests
> will have to unpack and start each assembly to run its set of tests.
> With four distinct sets of tests and three assemblies (same example),
> you will unpack, start and deploy on each assembly once for that set
> of tests.  A total of 12 unpacks, starts and deploys will be required.
>
> > This is how I'd imagined it, at a very high level.
> > - The itests subproject will have it's own set of modules. These are
> > the modules in G that have dropped their itests into this subproject.
>
> Totally possible.
>
> > - The tests will be grouped logically into suites or categories. Each
> > individual test will fall in one of these suites.
>
> Not sure of your usage of the word "test" in this sentence.  In junit
> terms is that TestSuite, TestCase, or individual test method.  Can
> you clarify?
>
> > - The tests can all be run or only a subset of suites can be run
> > quite easily.
>
> When you say "easily" do you mean by virtue of them being separate or
> some other idea?
>
> > - Continuum will continue to build G first and then run itests on the
> > assemblies.
> > - If the assembly passes a certain set of tests, then it can be
> > certified good and then be  uploaded to the repository.
>
> The maven 2 lifecycle supports this flow on a per module basis
> (truncated, but still in order):
>
>    compile
>    test-compile
>    test
>    package
>    before-integration-test
>    integration-test
>    after-integration-test
>    verify
>    install
>    deploy
>
> If there is a failure the next phase of the lifecycle is not called.
> So if "test" fails, "package" is not called.  If "integration-test"
> fails then "install" or "deploy" are not called.
>
>     Note, there are no "integration-test-complile" or "integration-
> test-package"
>     phases as it is expected these exist in other modules.
>
> The life cycle of assembly SuperServer could work in maven 2 as follows:
>
> Geronimo :: Assembly :: Super Server
>
>    compile        (no op)
>    test-compile   (no op)
>    test           (no op)
>    package        build the assembly
>    before-integration-test    unpack and start the server, deploy
> required itest components
>    integration-test           run all itests applicable for this
> assembly
>    after-integration-test     stop server
>    verify         run any additional checks to verify the package is
> valid and meets quality criteria.
>    install        put in local maven repo
>    deploy         upload to public repo
>
>
> If you wish to split out integration-test from the maven lifecycle
> and delay publishing the SuperServer binary, you would need at least
> three modules to achieve the same.
>
> Geronimo :: Assembly :: Super Server
>
>    compile        (no op)
>    test-compile   (no op)
>    test           (no op)
>    package        build the assembly
>    before-integration-test    (no op)
>    integration-test           (no op)
>    after-integration-test     (no op)
>    verify         (no op)
>    install        put in local maven repo
>    deploy         WARNING: don't call this our it will be upload to
> public repo without testing
>
> Geronimo :: iTest :: Super Server
>
>    compile        (no op)
>    test-compile   (no op)
>    test           (no op)
>    package        (no op)
>    before-integration-test    unpack and start the SuperServer,
> deploy required itest components
>    integration-test           run all itests applicable for this
> assembly
>    after-integration-test     stop the SuperServer
>    verify         run any additional checks to verify the package is
> valid and meets quality criteria.
>    install        (no op)
>    deploy         (no op)
>
> Geronimo :: Deploy :: Super Server
>
>    compile        (no op)
>    test-compile   (no op)
>    test           (no op)
>    package        (no op)
>    before-integration-test    (no op)
>    integration-test           (no op)
>    after-integration-test     (no op)
>    verify         devise some way to verify the itests have run
> successfully on this exact SuperServer binary
>    install        (no op)
>    deploy         upload to public repo
>
> You will have to do a lot of tricking on maven to do things like
> this, but we've proven that with enough hacking you can push maven
> against it's grain.
>
> > - a download page will be available to download assemblies and
> > binaries from every single build. The builds that passed itests will
> > be clearly marked so.
>
> Great.
>
>
> -David
>

Mime
View raw message