cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominic Williams <thedwilli...@googlemail.com>
Subject Re: Pelops - a new Java client library paradigm
Date Mon, 14 Jun 2010 15:15:55 GMT
Hi, re: pools and detecting node failure...

Pooling is handled by ThriftPool. This class maintains a separate
NodeContext object for each known node. This in turn maintains a pool of
connections to its node.

Each NodeContext has a single "poolRefiller" object/thread, which runs
either when signalled, or every ~2s, whichever is the sooner. Whenever it
runs, the first thing it does is check which of its existing pooled
connections are open. This is necessary for it to correctly calculate the
number of new connections to open (assuming it has to)

To check whether a connection is open, it calls TTransport.isOpen, which is
TSocket.isOpen, which is Socket.isConnected. If a connection is not open,
then it is binned.

Therefore, pretty quickly if a node has failed, the NodeContext will not be
holding any connections to it. This causes the NodeContext.isAvailable
method to return false. When this is the case, that node is not considered
by ThriftPool when it is seeking to return a connection to an operand
(Mutator, Selector, KeyDeletor etc object)

The pool refiller thread keeps on trying to create connections to a node,
even after all connections to it have failed. When/if it becomes available
again, then as soon as a connection is made NodeContext.isAvailable will
return true and it comes "back online" for the purposes of the operands.

NOTE: Some of my colleagues were working on Windows machines separated from
our local development servers by low-end NAT routers. After some period
using this Cassandra, inside Pelops even though TSocket.isOpen was returning
true, when an operand tried using connections they were getting a timeout or
other network exception. Calling setKeepAlive(true) on the underlying socket
does not prevent this (although this option is best set because in general
it should force timely detection of connection failure). Hector also
experienced similar problems and we adopt a similar response - by default
you'll see that Pelops sets Policy.getKillNodeConnsOnException() to true by
default. What this means is that if a network exception is thrown when an
operand interacts with a node, the NodeContext destroys all pooled
connections to that node on the basis that the general failure of
connections to that node may not be detectable because of the network setup.
Of course, not many people will be running their Cassandra clients from
Windows behind NAT in production, but the option is set by default because
otherwise a segment of developers trying the library will experience
persistent problems due to this network (and/or Thrift) strangeness and in
production we are ourselves will switch it off (although note the worse
downside is that the occasional network error to a node will cause the
refreshing of its pool)

Hope this makes sense.
Best, Dominic

On 14 June 2010 15:32, Kochheiser,Todd W - TO-DITT1 <twkochheiser@bpa.gov>wrote:

>  Great API that looks easy and intuitive to use.  Regarding your
> connection pool implementation, how does it handle failed/crashed nodes?
> Will the pool auto-detect failed nodes via a “tester” thread or will a
> failed node, and hence its pooled connection(s), be removed only when they
> are used?  Conversely, how will the pool be repopulated once the
> failed/crashed node becomes available?
>
>
>
> Todd
>
>
>  ------------------------------
>
> *From:* Dominic Williams [mailto:thedwilliams@googlemail.com]
> *Sent:* Friday, June 11, 2010 7:05 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Pelops - a new Java client library paradigm
>
>
>
> Hi good question.
>
>
>
> The scalability of Pelops is dependent on Cassandra, not the library
> itself. The library aims to provide an more effective access layer on top of
> the Thrift API.
>
>
>
> The library does perform connection pooling, and you can control the size
> of the pool and other parameters using a policy object. But connection
> pooling itself does not increase scalability, only efficiency.
>
>
>
> Hope this helps.
>
> BEst, Dominic
>
>
>
> On 11 June 2010 14:47, Ian Soboroff <isoboroff@gmail.com> wrote:
>
> Sounds nice.  Can you say something about the scales at which you've used
> this library?  Both write and read load?  Size of clusters and size of data?
>
> Ian
>
>
>
> On Fri, Jun 11, 2010 at 9:41 AM, Dominic Williams <
> thedwilliams@googlemail.com> wrote:
>
> Pelops is a new high quality Java client library for Cassandra.
>
>
>
> It has a design that:
>
> * reveals the full power of Cassandra through an elegant "Mutator and
> Selector" paradigm
>
> * generates better, cleaner, less bug prone code
>
> * reduces the learning curve for new users
>
> * drives rapid application development
>
> * encapsulates advanced pooling algorithms
>
>
>
> An article introducing Pelops can be found at
>
>
> http://ria101.wordpress.com/2010/06/11/pelops-the-beautiful-cassandra-database-client-for-java/
>
>
>
> Thanks for reading.
>
> Best, Dominic
>
>
>
>
>

Mime
View raw message