river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Venners <bv-...@artima.com>
Subject Re: Concurrency and River
Date Wed, 03 Oct 2007 00:06:53 GMT
Hi Dan,

On Oct 1, 2007, at 12:26 PM, Dan Creswell wrote:

> Hi Bill,
> I think your idea is an interesting one and certainly and I think the
> lessons learnt from Jini would be valuable in this area though I'm not
> sure that Jini in its current incarnation is the correct base to start
> from.  That of course might provide us with a target to evolve
> towards.....or perhaps we start a side-project for this work, who  
> knows?
I spoke with Allen Holub a couple hours ago about actors, and one of  
the things I learned was that actors don't really have a parallel  
processing story. Actors have little message queues, and they sit  
there processing and sending messages to other actors, so if you were  
to, say, model compute workers as actors, you'd still have the  
problems of dividing the work up and figuring out how to send jobs as  
messages to the worker actors, deal with failure, etc.

Back in 1993 I was trying to figure out how to use JUnit to write the  
ServiceUI conformance test kit, required by the JDP, and became  
frustrated. I ended up creating a test framework called SuiteRunner.  
I released it open source, but stopped at around version 0.8. As soon  
as I got it far enough to serve our purposes at Artima, I could see  
no value out of trying to promote it as an alternative to JUnit. One  
of the things I had imagined doing with it someday, if it ever made  
sense, was making it distribute test jobs via a Javaspaces compute  
grid. Unit tests are embarrassingly parallelizable, and people sit  
around waiting for them to run sequentially. With the rise of multi- 
core, this may start to seem pretty wasteful. A year or two after I  
released SuiteRunner to the public, Cedric Beust released TestNG, and  
his reasoning included many of the same frustrations I had had with  
JUnit. TestNG does have some kind of ability to distribute testing  
jobs to other programmers workstations, but doesn't seem to use Jini  
and I'm not sure how well his distribution strategy actually works.

I started porting SuiteRunner to Scala in part as a way to learn  
Scala and in part because we want to write tests in Scala at Artima.  
Also, since it is early in Scala history, I figured there's a chance  
SuiteRunner could become the de facto standard testing tool for Scala  
programmers. And since multi-core is coming, I thought if I made this  
new version of SuiteRunner ready for multi-core from the beginning,  
it might serve a real need people in the Java community will soon  
have that JUnit doesn't address.

To speed up unit testing, my idea has been that I keep a SuiteRunner  
server VM running at all time while developing, and when I run tests  
what I actually do is feed jobs to that server. This way I could  
avoid starting up a new JVM each time I want to run a test. So what  
seems to make sense is to embed a JavaSpace inside that SuiteRunner  

I actually wasn't thinking about SuiteRunner when I brought up the  
multi-core marketing opportunity a few days ago, but now I realize it  
may be a good example. The SuiteRunner server could look up how many  
CPUs are available and use that info to decide how many threads to  
fire up. My rough idea is that to run a suite of tests, a client  
would put a run job in the space. Some thread in the server would  
remove that job and partition it into suite jobs that go back into  
the space. Worker threads that remove suites from the space and  
execute the tests in them, sending reports to reporters. A reporter  
could put report entries into the space for the client to use, in  
some cases. The worker threads could use Jini discovery to find such  
spaces, so that if multiple programmers are sitting on the same  
subnet each running SuiteRunner servers, they could all use each  
other's spare CPU cycles to speed unit testing up.

The more programmers I could get to use this tool, the more  
programmers would "get" Jini and JavaSpaces I think. Because all Java  
programmers should really be doing unit testing. If it were the de  
facto standard in the Scala community, Scala programmers would reach  
for it by default. Java programmers might be convinced to use it so  
they can at least write unit tests in Scala, or take advantage of  
multiple cores or their coworkers spare compute cycles. It might be  
especially compelling for Java folks if it could distribute JUnit and  
TestNG tests too.

What do all of you think of this idea, and what advice would you have  
for me as far as what JavaSpace implementation to use? Blitz  
advertises that it can be embedded. Do you think that would be the  
best JavaSpace option for this use case?


Bill Venners
Artima, Inc.

View raw message