geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Blevins <david.blev...@visi.com>
Subject Re: Geronimo ORB progress
Date Wed, 16 Nov 2005 21:25:47 GMT
Sounds cool.

Just to give you some direct guidance, you probably should have shot  
an email out to the group saying "what do you guys do for testing  
when more than one VM is involved" or "we have this idea for testing  
in more than one VM, what do you think?"  Not a big deal, you're  
here, you want to do good work, we can work with that.

Here is the reply I would have given to the above questions.  I've  
Cc'ed James and Vincent as I know they are working on or have  
slightly similar things.

Sounds like a neat idea.  We have the itests plugin and the a bunch  
of tests that use the itest plugin.  We definitely want more.   
Ideally we'd have them for all the major compoments aside from just  
ejb, persistence, and rmi stuff.

The basic idea behind itests that makes them slightly different than  
plain junit tests ran by maven is that it's assumed that there are  
other systems (servers, databases, brokers, etc) that need to be  
started or connected to before the tests can run.  The tests  
themselves don't setup the other systems themselves so that the tests  
don't become coupled with them and you can actually swap out various  
aspects of those systems behind the back of the tests; server  
version, protocol implementation, java version, VM implementation,  
database provider, operating systems, even using embedded servers and  
embedded databases.

The idea actually evolved from the tests in OpenEJB that do this.  It  
took a lot of work to get them to run as part of the build and the  
itest plugin was the result.  So now we can easily boot up geronimo,  
derby, axion, activemq or whatever in whatever vm and on whatever OS  
and run the exact same tests to see how things are.

Definitely give those a look at:  http://cvs.openejb.org/viewrep/ 
openejb/openejb/modules/itests/src/java/org/openejb/test

Those particular tests allow you to plug in facades for the server  
and the database so the client (the tests) can say "give me an  
initial context" or "create these tables i need" in a generic way.   
It's assumed that those systems were setup and guaranteed in working  
state in the itest setup phase.  It's also guaranteed that the server  
and database facades know how to contact the server or database to  
carry out the "give me an initial context" and "create these tables"  
calls.

Here are some implementations of the database provider for reference:
   http://cvs.openejb.org/viewrep/openejb/openejb/modules/itests/src/ 
itest/org/openejb/test/AxionTestDatabase.java
   http://cvs.openejb.org/viewrep/openejb/openejb/modules/itests/src/ 
itest/org/openejb/test/DerbyTestDatabase.java
   http://cvs.openejb.org/viewrep/openejb/openejb/modules/itests/src/ 
itest/org/openejb/test/InstantDbTestDatabase.java
   http://cvs.openejb.org/viewrep/openejb/openejb/modules/itests/src/ 
itest/org/openejb/test/PostgreSqlTestDatabase.java

Here are some implementations for the server facades:
   http://cvs.openejb.org/viewrep/openejb/openejb/modules/itests/src/ 
itest/org/openejb/test/CorbaTestServer.java
   http://cvs.openejb.org/viewrep/openejb/openejb/modules/itests/src/ 
itest/org/openejb/test/RemoteTestServer.java
   http://cvs.openejb.org/viewrep/openejb/openejb/modules/itests/src/ 
itest/org/openejb/test/IvmTestServer.java
   http://cvs.openejb.org/viewrep/openejb/openejb/modules/itests/src/ 
itest/org/openejb/test/RiTestServer.java

Using the itest concept you could setup any system you need before  
hand, then just provide an abstraction of that system to the actual  
tests.  So it's not limited to just a server or a database.  You  
could do queue, topic, clustering heartbeat agent, directory server,  
etc.

Even with just what we have we can get quite far.  In a perfect  
world, we would actually test against the full matrix:

   Server: Local, Database: Axion
   Server: Local, Database: Derby
   Server: Local, Database: PostgreSQL
   Server: Remote, Database: Axion
   Server: Remote, Database: Derby
   Server: Remote, Database: PostgreSQL
   Server: Corba, Database: Axion
   Server: Corba, Database: Derby
   Server: Corba, Database: PostgreSQL

I had that going for a short while years ago but it was just too much  
infrastructure for me to keep running.  It would be nice to get  
Oracle and MySQL in that list as well.

In an even more perfect world we'd test against not just different  
Server and Database combinations, but JVM versions as well.

   Server: Local, Database: Axion, JVM: 1.3
   Server: Local, Database: Axion, JVM: 1.4
   Server: Local, Database: Axion, JVM: 1.5
   Server: Local, Database: Derby, JVM: 1.3
   Server: Local, Database: Derby, JVM: 1.4
   Server: Local, Database: Derby, JVM: 1.5
   Server: Local, Database: PostgreSQL, JVM: 1.3
   Server: Local, Database: PostgreSQL, JVM: 1.4
   Server: Local, Database: PostgreSQL, JVM: 1.5
   Server: Remote, Database: Axion, JVM: 1.3
   Server: Remote, Database: Axion, JVM: 1.4
   Server: Remote, Database: Axion, JVM: 1.5
   Server: Remote, Database: Derby, JVM: 1.3
   Server: Remote, Database: Derby, JVM: 1.4
   Server: Remote, Database: Derby, JVM: 1.5
   Server: Remote, Database: PostgreSQL, JVM: 1.3
   Server: Remote, Database: PostgreSQL, JVM: 1.4
   Server: Remote, Database: PostgreSQL, JVM: 1.5
   Server: Corba, Database: Axion, JVM: 1.3
   Server: Corba, Database: Axion, JVM: 1.4
   Server: Corba, Database: Axion, JVM: 1.5
   Server: Corba, Database: Derby, JVM: 1.3
   Server: Corba, Database: Derby, JVM: 1.4
   Server: Corba, Database: Derby, JVM: 1.5
   Server: Corba, Database: PostgreSQL, JVM: 1.3
   Server: Corba, Database: PostgreSQL, JVM: 1.4
   Server: Corba, Database: PostgreSQL, JVM: 1.5

If you add JVM vendors (Sun, IBM, Apple, BEA) to the list, the  
combinations goes up to like 109.  Throw on OS implementations and  
you get an insane number of test runs to complete.

Anyway, some of the this we still need:

   1) some nice way of specifying the items from the matrix to use  
(which vms, which databases, etc.)
   2) Some way to automate running through the matrix.
   3) Hardware (never can have too much)
   4) Licenses for the OSs and Databases

Hope that gives some insight to the lay of the land as it is now.   
Nothing is perfect or set in stone, so improvement is always welcome.

That said, the last part (the matrix) sounds very much like Cargo.   
Maybe Vincent can comment on that.
I also know James has also been trying to solve the matrix issue.   
Don't know what he's got going on.

-David

On Nov 16, 2005, at 6:15 AM, Anders Hessellund Jensen wrote:

> David Blevins wrote:
>> So what kind of testing framework is this? Is it junit-based?
>
> Yes, it is junit-based. Essentially, we have created a test  
> superclass, which inherits from junit.framework.Test.
>
> So, assume we would like a Multi-JVM test with a single test case  
> named "Simple". We have two JVM's, a client and a server. The  
> client and server each needs to do some initialization, then the  
> client initiates a test, and finally the server needs to do some  
> checks after the test.
>
> A multi-JVM junit test would look something like this:
>
> class MultiJvmTest extends RemoteTest {
>   public void setUp() {
>     // Starts a JVM named "server"
>     startTestAgent("server");
>     // Starts a JVM named "client"
>     startTestAgent("client");
>   }
>
>   public void serverBeforeSimple() {
>     // Setup the server for test case "Simple"
>     // This method will be invoked in the "server" JVM.
>   }
>   public void clientBeforeSimple() {
>     // Setup the client for test case "Simple".
>     // This method will be invoked in the "client" JVM.
>   }
>
>   public void clientTestSimple() {
>     // Run the test
>   }
>
>   public void serverAfterSimple() {
>     // Check server state after test.
>   }
> }
>
> When running this test, the framework would do the following:
>
> - run setUp(). This starts the two JVM's identified by the strings  
> "server" and "client". Thus, there is three JVM's running in total,  
> a master JVM, and two slave JVMs.
> - invoke the serverBeforeSimple() in the "server" JVM.
> - invoke the clientBeforeSimple() in the "client" JVM.
> - invoke the clientTestSimple() in the "client" JVM.
> - invoke the serverAfterSimple() in the "server" JVM.
>
> The whole thing integrates pretty well with junit. If an exception  
> is thrown in a slave JVM, it will be rethrown in the master JVM,  
> resulting in a correct stack trace identifying where the test  
> failed. Furthermore, it is possible to start an agent within the  
> master JVM, so single-step debugging is possible as well.
>
> Communication between the JVM's takes place using plain RMI.
>
> An agent can be given a Properties as argument. There is also a  
> global set of properties shared by all JVMs.
>
> The framework uses reflection to collect the methods that is called  
> when the test is run. The methods must be named:
>
> <agentName>Before<testName>
> <agentName>Test<testName>
> <agentName>After<testName>
>
>> How does it related to our existing itests?  Those are server  
>> (and  database) agnostic and can be ran across several vms and  
>> orbs.  Don't  know if there is any overlap.
>
> I don't know how the existing itests work. There may be some overlap.
>
>> Would these test be run as part of a normal build or would we set   
>> them up to run periodically in continuum or something?
>
> These tests are intended to be run as "normal" junit tests. Thus  
> they should be part of the normal build.
>
>> Are the tests focused on verifying corba compliance or is this   
>> somehow more generic and applicable to testing in general?
>
> The framework is quite general, but of course tailored with the  
> features we needed. There is nothing CORBA-specific in it.
>
> I would be happy to know if there is code out there that does  
> something similar to this framework. We have looked around, but  
> haven't really found anything that seemed quite right to our purpose.
>
> Anders
>


Mime
View raw message