river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Brouwer <mark.brou...@cheiron.org>
Subject Re: Concurrency and River
Date Thu, 04 Oct 2007 09:00:02 GMT
Bill Venners wrote:
> Hi Mark,
> On Oct 3, 2007, at 2:16 PM, Mark Brouwer wrote:
>>> To speed up unit testing, my idea has been that I keep a SuiteRunner 
>>> server VM running at all time while developing, and when I run tests 
>>> what I actually do is feed jobs to that server. This way I could 
>>> avoid starting up a new JVM each time I want to run a test. So what 
>>> seems to make sense is to embed a JavaSpace inside that SuiteRunner 
>>> server.
>> Assuming you keep the SuiteRunner JVM running and submit suites through
>> a JavaSpace you will likely run into what I think a lot of people also
>> consider the hard parts of working with Jini/RMI (and often overlooked)
>> namely that you must make sure that you will be able to provide as a
>> codebase annotation all classes your test suites will depend on.
>> Assuming people are able to do that you will likely run into the fact
>> the JVM that is about to (re)execute the testsuites has those JAR files
>> cached (you can also prevent from using JAR files though).
> Wouldn't the JVM not cache JARs if the HTTP response header says 
> cache-control: no cache?

No, at least the default jar: protocol handler for the Sun JRE has a JAR
file cache works the way I mentioned, for the IBM JRE as well, which
makes me believe that this is true for every Java licensee.

When a class loader requires a JAR file to obtain the class definition
or some other resource it will go through the JAR protocol handler, with
an exception I deal with later, and when the JAR file has been
downloaded once it will reuse the cached (local) JAR file.

This behavior can be understood from the class loader perspective in
which you don't want to download a remote JAR file for each time you
want to obtain a class definition and for which consistency of classes
relative to each other is important. The default jar: protocol handler
and cache logic ignores cache-control directives and performs no checks
for If-Modified-Since.

This is exactly the reason why in many application servers you often run
into problems with the application server not seeing updated classes
when they act as a Jini client. Although I won't go to deep into the
details and don't want to make a marketing pitch but Seven provides a
platform that does solve a lot of these things by deploying a jar:
protocol handler for which there is a 2 layer cache in which one layer
of the cache is bound to the class loader (and any child class loaders)
that uses cached JAR files from a common JAR file pool that is populated
with all JAR files to be downloaded and that keeps track of different
versions (check If-Modified-Since, etc.). When a new class loader tree
is created it will see the latest consistent collection of JAR files
even while other class loader might see an older but also consistent
view of the JAR files. This cache is also persistent in that it can
survive JVM restarts so in case the mobile code is no longer available
over the network you can still continue to work as if it is there (this
is required to make your services/clients less dependent on the
availability of mobile code). Also it provides in immense speedup in
case you have minimal bandwidth such as over the Internet or when you
have only a T1 connection to your data center.

The jar: protocol handler could be modified to work in other
environments although for various reasons recently I went for a tighter
integration to the class loader creation logic and container provided
persistence. But principles and techniques can be applied for any
container. It is however not possible to do this is in a generic way
(that I know of) for the Java Platform (otherwise I think it would have
been done I guess) for it is closely tied to the life-cycle of the
components you deploy in your environment.

>> There are ways to solve some of these problems by making sure you end up
>> with a different codebase annotation each time or configuring e.g. a
>> smart jar: protocol handler (you can find one in the Seven source code)
>> for your JVM that doesn't caches JAR files but is able to verify
>> whether a newer JAR file is available and will use the newer JAR files
>> for new class loaders created.
> I suppose there could also be a local mode in which classes are loaded 
> off the local disk through a new URLClassLoader created each run. This 
> is what SuiteRunner does now. This is probably what most developers 
> would want to do at their workstations.

Correct, in local mode you don't have that particular problem because
the URLClassLoader doesn't use the jar: protocol handler logic when it
detects it is dealing with a file URL. In that case it will perform some
optimizations and use the JAR file directly. Note that this is an
implementation detail of the current URLClassLoader implementations I
found in the field.

However on Windows you might run into another nasty problem when you
have a long running JVM. When JAR files are used by URLClassLoader on
Windows a file lock is obtained by the JVM that will only be released
when the JVM exits. The consequence of this is that while your JVM is
running you won't be able to overwrite the JAR files by e.g. your build
process on Windows.

For this reason Seven transforms all URLs that have the file protocol
and that are used for server side class loaders into a custom dfile
protocol so that this optimization by URLClassLoader doesn't take place
and when I dispose a class loader (explicitly or through GC) I
explicitly close the underlying JAR files.

As you can see proper handling of class evolution is a very painful
exercise and my experience is that most environments don't bring much
help to their developers. I know of people that work with Seven in
development in which they have their containers running for weeks (even
months) and in which they have upgraded various services in the
container hundreds of times and that they see updates made to mobile
code of other services in the network (no Seven deployed Jini services).

SuiteRunner in the typical case is started each time for running a set
of test suites, in your posting you indicate you want to have it running
as long lasting process. I think you should be aware of these cases then.

It can be sorted out, but it is a complicated matter, but principles I
applied in the Jini Service Container can be applied of course in any
other framework/platform. For more information and pointers into the
code feel free to ask questions.

> I'd forgotten about that. Hmm. Since you kind of made a little fork, 
> would it be alright with you if I didn't make new releases of the Java 
> SuiteRunner going forward?

That is OK with me, you were so kind to relicense to ALv2 just for me so
I can evolve when needed.

View raw message