Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 94635 invoked from network); 14 Jun 2010 15:16:24 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 14 Jun 2010 15:16:24 -0000 Received: (qmail 62120 invoked by uid 500); 14 Jun 2010 15:16:22 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 62083 invoked by uid 500); 14 Jun 2010 15:16:22 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 62075 invoked by uid 99); 14 Jun 2010 15:16:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 15:16:22 +0000 X-ASF-Spam-Status: No, hits=2.1 required=10.0 tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of thedwilliams@googlemail.com designates 209.85.161.44 as permitted sender) Received: from [209.85.161.44] (HELO mail-fx0-f44.google.com) (209.85.161.44) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 15:16:17 +0000 Received: by fxm13 with SMTP id 13so2766395fxm.31 for ; Mon, 14 Jun 2010 08:15:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=dN/a28ZCUpexQ+BFFrGqXwC9dwoHXsDXdSiQYf+EDxM=; b=bnULveTHFA0k9wuP/bX8f5banHkVL4llLNSsM9PBDYC4A8YF7gQyd7IyTLSQ2WfwFW TwFaWzaOdxLzEbfZcQqXRaUdZZv8M5AWBiRYip3jvpPlFSqVBNQ4wd+3dWuZlkPvPjvD WfaDogC5gdOBlIEMF2HKTRHtxQMUJJlhqlBWw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=kfJV98psvFrjS+x7uxYIy9e4ZeBrxLhtZ1a7/sIEYoZA4x8QzPKs6Czu9EgZYajR0G YyQW/zY8hB/7Gwlu4xZOz0BVmgqOM+LVJJfhR1e7xHax3G8+mkSNdlw64Bceb7ZYFlB5 4H1/GfPYkmcx5p5DbbRLOh1ub9n/AMH97S694= MIME-Version: 1.0 Received: by 10.239.159.148 with SMTP id y20mr415070hbc.42.1276528555553; Mon, 14 Jun 2010 08:15:55 -0700 (PDT) Received: by 10.239.160.129 with HTTP; Mon, 14 Jun 2010 08:15:55 -0700 (PDT) In-Reply-To: <2B4B52565669304C979DC60E52AA3F9B022C790130@spnvm1183.bud.bpa.gov> References: <2B4B52565669304C979DC60E52AA3F9B022C790130@spnvm1183.bud.bpa.gov> Date: Mon, 14 Jun 2010 16:15:55 +0100 Message-ID: Subject: Re: Pelops - a new Java client library paradigm From: Dominic Williams To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001485f27fae1974340488fef74f --001485f27fae1974340488fef74f Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi, re: pools and detecting node failure... Pooling is handled by ThriftPool. This class maintains a separate NodeContext object for each known node. This in turn maintains a pool of connections to its node. Each NodeContext has a single "poolRefiller" object/thread, which runs either when signalled, or every ~2s, whichever is the sooner. Whenever it runs, the first thing it does is check which of its existing pooled connections are open. This is necessary for it to correctly calculate the number of new connections to open (assuming it has to) To check whether a connection is open, it calls TTransport.isOpen, which is TSocket.isOpen, which is Socket.isConnected. If a connection is not open, then it is binned. Therefore, pretty quickly if a node has failed, the NodeContext will not be holding any connections to it. This causes the NodeContext.isAvailable method to return false. When this is the case, that node is not considered by ThriftPool when it is seeking to return a connection to an operand (Mutator, Selector, KeyDeletor etc object) The pool refiller thread keeps on trying to create connections to a node, even after all connections to it have failed. When/if it becomes available again, then as soon as a connection is made NodeContext.isAvailable will return true and it comes "back online" for the purposes of the operands. NOTE: Some of my colleagues were working on Windows machines separated from our local development servers by low-end NAT routers. After some period using this Cassandra, inside Pelops even though TSocket.isOpen was returnin= g true, when an operand tried using connections they were getting a timeout o= r other network exception. Calling setKeepAlive(true) on the underlying socke= t does not prevent this (although this option is best set because in general it should force timely detection of connection failure). Hector also experienced similar problems and we adopt a similar response - by default you'll see that Pelops sets Policy.getKillNodeConnsOnException() to true by default. What this means is that if a network exception is thrown when an operand interacts with a node, the NodeContext destroys all pooled connections to that node on the basis that the general failure of connections to that node may not be detectable because of the network setup= . Of course, not many people will be running their Cassandra clients from Windows behind NAT in production, but the option is set by default because otherwise a segment of developers trying the library will experience persistent problems due to this network (and/or Thrift) strangeness and in production we are ourselves will switch it off (although note the worse downside is that the occasional network error to a node will cause the refreshing of its pool) Hope this makes sense. Best, Dominic On 14 June 2010 15:32, Kochheiser,Todd W - TO-DITT1 w= rote: > Great API that looks easy and intuitive to use. Regarding your > connection pool implementation, how does it handle failed/crashed nodes? > Will the pool auto-detect failed nodes via a =93tester=94 thread or will = a > failed node, and hence its pooled connection(s), be removed only when the= y > are used? Conversely, how will the pool be repopulated once the > failed/crashed node becomes available? > > > > Todd > > > ------------------------------ > > *From:* Dominic Williams [mailto:thedwilliams@googlemail.com] > *Sent:* Friday, June 11, 2010 7:05 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Pelops - a new Java client library paradigm > > > > Hi good question. > > > > The scalability of Pelops is dependent on Cassandra, not the library > itself. The library aims to provide an more effective access layer on top= of > the Thrift API. > > > > The library does perform connection pooling, and you can control the size > of the pool and other parameters using a policy object. But connection > pooling itself does not increase scalability, only efficiency. > > > > Hope this helps. > > BEst, Dominic > > > > On 11 June 2010 14:47, Ian Soboroff wrote: > > Sounds nice. Can you say something about the scales at which you've used > this library? Both write and read load? Size of clusters and size of da= ta? > > Ian > > > > On Fri, Jun 11, 2010 at 9:41 AM, Dominic Williams < > thedwilliams@googlemail.com> wrote: > > Pelops is a new high quality Java client library for Cassandra. > > > > It has a design that: > > * reveals the full power of Cassandra through an elegant "Mutator and > Selector" paradigm > > * generates better, cleaner, less bug prone code > > * reduces the learning curve for new users > > * drives rapid application development > > * encapsulates advanced pooling algorithms > > > > An article introducing Pelops can be found at > > > http://ria101.wordpress.com/2010/06/11/pelops-the-beautiful-cassandra-dat= abase-client-for-java/ > > > > Thanks for reading. > > Best, Dominic > > > > > --001485f27fae1974340488fef74f Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Hi, re: pools and detecting node failure...

P= ooling is handled by ThriftPool. This class maintains a separate NodeContex= t object for each known node. This in turn maintains a pool of connections = to its node.

Each NodeContext has a single "poolRefiller" = object/thread, which runs either when signalled, or every ~2s, whichever is= the sooner. Whenever it runs, the first thing it does is check which of it= s existing pooled connections are open. This is necessary for it to correct= ly calculate the number of new connections to open (assuming it has to)

To check whether a connection is open, it calls TTransp= ort.isOpen, which is TSocket.isOpen, which is Socket.isConnected. If a conn= ection is not open, then it is binned.

Therefore, = pretty quickly if a node has failed, the NodeContext will not be holding an= y connections to it. This causes the NodeContext.isAvailable method to retu= rn false. When this is the case, that node is not considered by ThriftPool = when it is seeking to return a connection to an operand (Mutator, Selector,= KeyDeletor etc object)

The pool refiller thread keeps on trying to create conn= ections to a node, even after all connections to it have failed. When/if it= becomes available again, then as soon as a connection is made NodeContext.= isAvailable will return true and it comes "back online" for the p= urposes of the operands.

NOTE: Some of my=A0colleagues=A0were working on Windows= machines separated from our local development servers by low-end NAT route= rs. After some period using this Cassandra, inside Pelops even though TSock= et.isOpen was returning true, when an operand tried using connections they = were getting a timeout or other network exception. Calling setKeepAlive(tru= e) on the underlying socket does not prevent this (although this option is = best set because in general it should force timely detection of connection = failure). Hector also experienced similar problems and we adopt a similar r= esponse - by default you'll see that Pelops sets Policy.getKillNodeConn= sOnException() to true by default. What this means is that if a network exc= eption is thrown when an operand interacts with a node, the NodeContext des= troys all pooled connections to that node on the basis that the general fai= lure of connections to that node may not be detectable because of the netwo= rk setup. Of course, not many people will be running their Cassandra client= s from Windows behind NAT in production, but the option is set by default b= ecause otherwise a segment of developers trying the library will experience= persistent problems due to this network (and/or Thrift) strangeness and in= production we are ourselves will switch it off (although note the worse do= wnside is that the occasional network error to a node will cause the refres= hing of its pool)

Hope this makes sense.
Best, Dominic

On 14 June 2010 15:32, Kochheiser,Todd W -= TO-DITT1 <twk= ochheiser@bpa.gov> wrote:

Great API that loo= ks easy and intuitive to use.=A0 Regarding your connection pool implementation, how does it handle failed/crashed nodes?=A0 Will the pool auto-detect failed nodes via a =93te= ster=94 thread or will a failed node, and hence its pooled connection(s), be remove= d only when they are used?=A0 Conversely, how will the pool be repopulated once the failed/crashed node becomes available?

=A0<= /p>

Todd=A0

=A0<= /p>

From: Dominic Williams [mailto:th= edwilliams@googlemail.com]
Sent: Friday, June 11, 2010 = 7:05 AM
To: user@cassandra.apache.org
Subject: Re: Pelops - a new = Java client library paradigm

=A0

Hi good question.

=A0

The scalability of Pelops is dependent on Cassandra,= not the library itself. The library aims to provide an more effective access layer on top o= f the Thrift API.

=A0

The library does perform connection pooling, and you= can control the size of the pool and other parameters using a policy object. But connection pooling itself does not increase scalability, only efficiency.=A0

=A0

Hope this helps.

BEst, Dominic

=A0

On 11 June 2010 14:47, Ian Soboroff <isoboroff@gmail.com> wr= ote:

Sounds nice.=A0 Can you say something about the scal= es at which you've used this library?=A0 Both write and read load?=A0 Size of clusters and size of data?

Ian

=A0

On Fri, Jun 11, 2010 at 9:41 AM, Dominic Williams &l= t;thedwill= iams@googlemail.com> wrote:

Pelops is a new high quality Java client library for= Cassandra.

=A0

It has a design that:

* reveals the full power of Cassandra through an ele= gant "Mutator and Selector" paradigm

* generates better, cleaner, less bug prone code

* reduces the learning curve for new users

* drives rapid application development=

* encapsulates advanced pooling algorithms

=A0

An article introducing Pelops can be found at=

=A0

Thanks for reading.

Best, Dominic

=A0

=A0


--001485f27fae1974340488fef74f--