geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Strachan <>
Subject Re: Infiniband
Date Sun, 15 Jan 2006 14:17:59 GMT
On 14 Jan 2006, at 22:27, lichtner wrote:
> On Fri, 13 Jan 2006, James Strachan wrote:
>>> The infiniband transport would be native code, so you could use JNI.
>>> However, it would definitely be worth it.
>> Agreed! I'd *love* a Java API to Infiniband! Have wanted one for ages
>> & google every once in a while to see if one shows up :)
>> It looks like MPI has support for Infiniband; would it be worth
>> trying to wrap that in JNI?
> I did find that HP has a Java interface for MPI. However, to me it  
> doesn't
> necessarily seem that this is the way to go. I think for writing
> distributed computations it would be the right choice, but I think  
> that
> the people who write those choose to work in a natively compiled  
> language,
> and I think that this may be the reason why this Java mpi doesn't  
> appear
> to be that well-known.
> However I did find something which might work for us, namely UDAPL
> from the DAT Collaborative. A consortium created a spec for  
> interface to
> anything that provides RDMA capabilities:
> The header files and the spec are right there.
> I downloaded the only release made by and they claim
> that it only works with kernel 2.4, and that for 2.6 you have to use
> The latter claims to provide an implementation of UDAPL:
> The wiki has a lot of info.
> From the mailing list archive you can tell that this project has a  
> lot of
> momentum:

Awesome! Thanks for all the links

> I think the next thing to do would be to prove that using RDMA as  
> opposed
> to udp is worthwhile. I think it is, because JITs are so fast now,  
> but I
> think that before planning anything long-term I would get two
> infiniband-enabled boxes and write a little prototype.

Agreed; the important issue is gonna be, can Java with JNI (or Unsafe  
or one of the alternatives to JNI: 
2005/07/28/AlternativestoJavaNativeI.html) work with RDMA using  
native ByteBuffers so that the zero copy is avoided and so that  
things perform better than just using some Infiniband-optimised TCP/ 
IP implementation. Though to be able to test this we need to make a  
prototype Java API to RDMA :) But it is definitely well worth the  
experiment IMHO

The main big win is obviously avoiding the double copy of working  
with TCP/IP though there are other benefits like improved flow  
control (you know that you can send a message to a consumer & how  
much capacity it has at any point in time so there is no need to  
worry about slow/dead consumer detection) another is concurrency; in  
a point-cast model, sending to multiple consumers in 1 thread is  
trivial (and multi-threading definitely slows down messaging as we  
found in ActiveMQ).

In ActiveMQ the use of RDMA would allow us to do straight through  
processing for messages which could dramatically cut down on the  
number of objects created per message & the GC overhead. (Indeed  
we've been musing that it might be possible to avoid most per-message  
dispatch object allocations if selectors are not used and we wrote a  
custom RDMA friendly version of ActiveMQMessage; we should also be  
able to optimise the use of the Journal as we can just pass around  
the ByteBuffer rather than using OpenWire marshalling.

> I think Appro sells
> infiniband blades with Mellanox hcas.
> There is also IBM's proprietary API for clustering mainframes, the
> Coupling Facility:
> There are some amazing articles there.

Great stuff - thanks for the link!

> Personally I also think there is value in implementing replication  
> using
> udp (process groups libraries such as evs4j), so I would pursue  
> both at
> the same time.

Yeah; like many things in distributed systems & messaging; it all  
depends on what you are doing as to what solution is the best for  
your scenario. Certainly both technologies are useful tools to have  
in your bag when creating middleware. I personally see RDMA as a  
possible faster alternative for TCP/IP inside message brokers such as  
ActiveMQ as well as for request-response messaging such as in openejb.


To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre.

View raw message