cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Molinaro <antho...@alumni.caltech.edu>
Subject Re: Cassandra & HAProxy
Date Sun, 29 Aug 2010 18:04:22 GMT

On Sat, Aug 28, 2010 at 02:44:41PM -0700, Benjamin Black wrote:
> On Sat, Aug 28, 2010 at 2:34 PM, Anthony Molinaro
> <anthonym@alumni.caltech.edu> wrote:
> > I think maybe he thought you meant put a layer between cassandra internal
> > communication.
> 
> No, I took the question to be about client connections.

Sorry didn't mean to put words into your mouth

> > There's no problem balancing client connections with
> > haproxy, we've been pushing several billion requests per month through
> > haproxy to cassandra.
> >
> 
> Can it be done: yes.  Is it best practice: no.  Even 10 billion
> requests/month is an average of less than 4000 reqs/sec.   Just not
> that many for a distributed database like Cassandra.

I don't know it seems to tax our setup of 39 extra large ec2 nodes, its
also closer to 24000 reqs/sec at peak since there are different tables
(2 tables for each read and 2 for each write)

> Cassandra can, and does, fail in ways that do not stop it from
> answering TCP connection requests.  Are you saying it works fine
> because you have seen numerous types of node failures and this was
> sufficient? I would be quite surprised if that were so.  Using an LB
> for service discovery is a fine thing (connect to a VIP, call
> describe_ring, open direct connections to cluster nodes).  Relying on
> an LB to do the right thing when it is totally ignorant of what is
> going across those client connections (as is implied by simply
> checking for connectivity) is asking for trouble.  Doubly so when you
> use a leastconn policy (a failing node can spit out an error and close
> a connection with impressive speed, sucking all the traffic to itself;
> common problem with HTTP servers giving back errors).


The haproxy does seem sufficient for us.  We've been running with cassandra
in production since 0.3.0 and seen just about every possible failure.  For
the most part it has worked.  I'm not saying it's the most efficient, just
that it will work for most people's usage.  All the writes to this cluster
are via php, which creates a connection for each request, so a connection
check works fine in this case.  We attempt to pool connections via java for
reads, but they reconnect whenever they receive an error.

If one machine is misbehaving it tends to fail pretty quickly, at which 
point all the haproxies drop it (we have an haproxy on every client node,
so it acts like a connection pooling mechanism for the client).  describe_ring
is a newish call, it didn't exist when we wrote our systems and we have not
had a chance to revisit.  So while yes there are problems with using an
haproxy, they are not insurmountable, and it would probably work for many
use cases.  But like everything YMMV.

-Anthony

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@alumni.caltech.edu>

Mime
View raw message