incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kochheiser,Todd W - TO-DITT1" <>
Subject RE: Pelops - a new Java client library paradigm
Date Mon, 14 Jun 2010 19:27:37 GMT
Thank you for the very clear and detailed explanation of how the pool works.   I think I'll
give Pelops a try.


From: Dominic Williams []
Sent: Monday, June 14, 2010 8:16 AM
Subject: Re: Pelops - a new Java client library paradigm

Hi, re: pools and detecting node failure...

Pooling is handled by ThriftPool. This class maintains a separate NodeContext object for each
known node. This in turn maintains a pool of connections to its node.

Each NodeContext has a single "poolRefiller" object/thread, which runs either when signalled,
or every ~2s, whichever is the sooner. Whenever it runs, the first thing it does is check
which of its existing pooled connections are open. This is necessary for it to correctly calculate
the number of new connections to open (assuming it has to)

To check whether a connection is open, it calls TTransport.isOpen, which is TSocket.isOpen,
which is Socket.isConnected. If a connection is not open, then it is binned.

Therefore, pretty quickly if a node has failed, the NodeContext will not be holding any connections
to it. This causes the NodeContext.isAvailable method to return false. When this is the case,
that node is not considered by ThriftPool when it is seeking to return a connection to an
operand (Mutator, Selector, KeyDeletor etc object)

The pool refiller thread keeps on trying to create connections to a node, even after all connections
to it have failed. When/if it becomes available again, then as soon as a connection is made
NodeContext.isAvailable will return true and it comes "back online" for the purposes of the

NOTE: Some of my colleagues were working on Windows machines separated from our local development
servers by low-end NAT routers. After some period using this Cassandra, inside Pelops even
though TSocket.isOpen was returning true, when an operand tried using connections they were
getting a timeout or other network exception. Calling setKeepAlive(true) on the underlying
socket does not prevent this (although this option is best set because in general it should
force timely detection of connection failure). Hector also experienced similar problems and
we adopt a similar response - by default you'll see that Pelops sets Policy.getKillNodeConnsOnException()
to true by default. What this means is that if a network exception is thrown when an operand
interacts with a node, the NodeContext destroys all pooled connections to that node on the
basis that the general failure of connections to that node may not be detectable because of
the network setup. Of course, not many people will be running their Cassandra clients from
Windows behind NAT in production, but the option is set by default because otherwise a segment
of developers trying the library will experience persistent problems due to this network (and/or
Thrift) strangeness and in production we are ourselves will switch it off (although note the
worse downside is that the occasional network error to a node will cause the refreshing of
its pool)

Hope this makes sense.
Best, Dominic

On 14 June 2010 15:32, Kochheiser,Todd W - TO-DITT1 <<>>
Great API that looks easy and intuitive to use.  Regarding your connection pool implementation,
how does it handle failed/crashed nodes?  Will the pool auto-detect failed nodes via a "tester"
thread or will a failed node, and hence its pooled connection(s), be removed only when they
are used?  Conversely, how will the pool be repopulated once the failed/crashed node becomes


From: Dominic Williams [<>]
Sent: Friday, June 11, 2010 7:05 AM
Subject: Re: Pelops - a new Java client library paradigm

Hi good question.

The scalability of Pelops is dependent on Cassandra, not the library itself. The library aims
to provide an more effective access layer on top of the Thrift API.

The library does perform connection pooling, and you can control the size of the pool and
other parameters using a policy object. But connection pooling itself does not increase scalability,
only efficiency.

Hope this helps.
BEst, Dominic

On 11 June 2010 14:47, Ian Soboroff <<>>
Sounds nice.  Can you say something about the scales at which you've used this library?  Both
write and read load?  Size of clusters and size of data?


On Fri, Jun 11, 2010 at 9:41 AM, Dominic Williams <<>>
Pelops is a new high quality Java client library for Cassandra.

It has a design that:
* reveals the full power of Cassandra through an elegant "Mutator and Selector" paradigm
* generates better, cleaner, less bug prone code
* reduces the learning curve for new users
* drives rapid application development
* encapsulates advanced pooling algorithms

An article introducing Pelops can be found at

Thanks for reading.
Best, Dominic

View raw message