Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 45831 invoked from network); 30 Aug 2010 17:03:07 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 30 Aug 2010 17:03:07 -0000 Received: (qmail 73256 invoked by uid 500); 30 Aug 2010 17:03:05 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 73210 invoked by uid 500); 30 Aug 2010 17:03:04 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 73202 invoked by uid 99); 30 Aug 2010 17:03:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 17:03:04 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of daveviner@gmail.com designates 209.85.161.44 as permitted sender) Received: from [209.85.161.44] (HELO mail-fx0-f44.google.com) (209.85.161.44) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 17:03:00 +0000 Received: by fxm18 with SMTP id 18so3893716fxm.31 for ; Mon, 30 Aug 2010 10:02:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=H5dmIxDpXpLQllF8o2QnTaIA60D6xnbpF4C2hpIE6gI=; b=h3R9nsWqzjgwLNRLpxib4Y9SBOvbyxRrxfaYaRq9Z5fzPxXlPq6nk/ToTlmdkuE132 AuVri9Q0RtqtszAoBac8d27lkIsBwh1zl28UBkM/0IrdgrWnPejG/avc7du6ofgb6aQt AtDV5CXSAvuRs4Seng4tjAirpo+aTuXMIOLlU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; b=KQpsWYaH/2yFm3MROtq7m0KKJXqtrWoBEaoIP10+/XmdSZ09h5yb2t5CjXjVCRqWZU CeA0vQVyGjXhVcgAR7aKNBBo/vyWMqRCpfPRGOIPDHXBxf+jFXqXgasO+ibT0JAMkHeU oc677XwBBb/3Izu/TnBIUu1L+Ru6U7/vCl61w= MIME-Version: 1.0 Received: by 10.223.119.10 with SMTP id x10mr4211235faq.1.1283187757087; Mon, 30 Aug 2010 10:02:37 -0700 (PDT) Sender: daveviner@gmail.com Received: by 10.223.18.216 with HTTP; Mon, 30 Aug 2010 10:02:36 -0700 (PDT) In-Reply-To: References: <4C794746.8040708@gmail.com> <4C79550E.1080907@gmail.com> <20100828213456.GA23023@alumni.caltech.edu> <20100829180422.GB23023@alumni.caltech.edu> <20100829234818.GH23023@alumni.caltech.edu> Date: Mon, 30 Aug 2010 10:02:36 -0700 X-Google-Sender-Auth: co5mHHNfYZR9A_Tq5MDNiFLmKxc Message-ID: Subject: Re: Cassandra & HAProxy From: Dave Viner To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001636c5a57971040b048f0d6edf --001636c5a57971040b048f0d6edf Content-Type: text/plain; charset=ISO-8859-1 Hi Edward, By "down hard", I assume you mean that the machine is no longer responding on the cassandra thrift port. That makes sense (and in fact is what I'm doing currently). But, it seems like the real improvement is something that would allow for a simple monitor that goes beyond the simple "machine not reachable" issue and covers more common scenarios that temporarily impact service time, but aren't so drastic as to cause machine outage. Dave Viner On Mon, Aug 30, 2010 at 9:52 AM, Edward Capriolo wrote: > On Mon, Aug 30, 2010 at 12:40 PM, Dave Viner wrote: > > FWIW - we've been using HAProxy in front of a cassandra cluster in > > production and haven't run into any problems yet. It sounds like our > > cluster is tiny in comparison to Anthony M's cluster. But I just wanted > to > > mentioned that others out there are doing the same. > > One thing in this thread that I thought was interesting is Ben's initial > > comment "the presence of the proxy precludes clients properly backing off > > from nodes returning errors." I think it would be very cool if someone > > implemented a mechanism for haproxy to detect the error nodes and then > > enable it to drop those nodes from the rotation. I'd be happy to help > with > > this, as I know how it works with haproxy and standard web servers or > other > > tcp servers. But, I'm not sure how to make it work with Cassandra, > since, > > as Ben points out, it can return valid tcp responses (that say > > "error-condition") on the standard port. > > Dave Viner > > > > On Sun, Aug 29, 2010 at 4:48 PM, Anthony Molinaro > > wrote: > >> > >> On Sun, Aug 29, 2010 at 12:20:10PM -0700, Benjamin Black wrote: > >> > On Sun, Aug 29, 2010 at 11:04 AM, Anthony Molinaro > >> > wrote: > >> > > > >> > > > >> > > I don't know it seems to tax our setup of 39 extra large ec2 nodes, > >> > > its > >> > > also closer to 24000 reqs/sec at peak since there are different > tables > >> > > (2 tables for each read and 2 for each write) > >> > > > >> > > >> > Could you clarify what you mean here? On the face of it, this > >> > performance seems really poor given the number and size of nodes. > >> > >> As you say I would expect to achieve much better performance given the > >> node > >> size, but if you go back and look through some of the issues we've seen > >> over time, you'll find we've been hit with nodes being too small, having > >> too few nodes to deal with request volume, having OOMs, having bad > >> sstables, > >> having the ring appear different to different nodes, and several other > >> problems. > >> > >> Many of i/o problems presented themselves as MessageDeserializer pool > >> backups > >> (although we stopped having these since Jonathan was by and suggested > row > >> cache of about 1Gb, thanks Riptano!). We currently have mystery OOMs > >> which are probably caused by GC storms during compactions (although > >> usually > >> the nodes restart and compact fine, so who knows). I also regularly > watch > >> nodes go away for 30 seconds or so (logs show node goes dead, then comes > >> back to life a few seconds later). > >> > >> I've sort of given up worrying about these, as we are in the process of > >> moving this cluster to our own machines in a colo, so I figure I should > >> wait until they are moved, and see how the new machines do before I > worry > >> more about performance. > >> > >> -Anthony > >> > >> -- > >> ------------------------------------------------------------------------ > >> Anthony Molinaro > > > > > > > Any proxy with a TCP health check should be able to determine if the > Cassandra service is down hard. The problem for the tools that are not > cassandra protocol aware are detecting slowness or other anomalies > like TimedOut exceptions. > > If you are seeing GC storms during compactions you might have rows > that are too big. When the compaction hits these memory spikes. I > lowered the compaction priority (and added more nodes) which has > helped compaction back off leaving some IO for requests. > --001636c5a57971040b048f0d6edf Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Edward,

By "down hard", I assume you mean t= hat the machine is no longer responding on the cassandra thrift port. =A0Th= at makes sense (and in fact is what I'm doing currently). =A0But, it se= ems like the real improvement is something that would allow for a simple mo= nitor that goes beyond the simple "machine not reachable" issue a= nd covers more common scenarios that temporarily impact service time, but a= ren't so drastic as to cause machine outage.

Dave Viner


On Mon, Aug 30, 2010 at 9:52 AM, Edward Capriolo <edlinuxguru@gmail.com> = wrote:
On Mon, A= ug 30, 2010 at 12:40 PM, Dave Viner <daveviner@pobox.com> wrote:
> FWIW - we've been using HAProxy in front of a cassandra cluster in=
> production and haven't run into any problems yet. =A0It sounds lik= e our
> cluster is tiny in comparison to Anthony M's cluster. =A0But I jus= t wanted to
> mentioned that others out there are doing the same.
> One thing in this thread that I thought was interesting is Ben's i= nitial
> comment "the=A0presence of the proxy precludes clients properly b= acking off
> from=A0nodes returning errors." =A0I think it would be very cool = if someone
> implemented a mechanism for haproxy to detect the error nodes and then=
> enable it to drop those nodes from the rotation. =A0I'd be happy t= o help with
> this, as I know how it works with haproxy and standard web servers or = other
> tcp servers. =A0But, I'm not sure how to make it work with Cassand= ra, since,
> as Ben points out, it can return valid tcp responses (that say
> "error-condition") on the standard port.
> Dave Viner
>
> On Sun, Aug 29, 2010 at 4:48 PM, Anthony Molinaro
> <anthonym@alumni.cal= tech.edu> wrote:
>>
>> On Sun, Aug 29, 2010 at 12:20:10PM -0700, Benjamin Black wrote: >> > On Sun, Aug 29, 2010 at 11:04 AM, Anthony Molinaro
>> > <anthonym@a= lumni.caltech.edu> wrote:
>> > >
>> > >
>> > > I don't know it seems to tax our setup of 39 extra l= arge ec2 nodes,
>> > > its
>> > > also closer to 24000 reqs/sec at peak since there are di= fferent tables
>> > > (2 tables for each read and 2 for each write)
>> > >
>> >
>> > Could you clarify what you mean here? =A0On the face of it, t= his
>> > performance seems really poor given the number and size of no= des.
>>
>> As you say I would expect to achieve much better performance given= the
>> node
>> size, but if you go back and look through some of the issues we= 9;ve seen
>> over time, you'll find we've been hit with nodes being too= small, having
>> too few nodes to deal with request volume, having OOMs, having bad=
>> sstables,
>> having the ring appear different to different nodes, and several o= ther
>> problems.
>>
>> Many of i/o problems presented themselves as MessageDeserializer p= ool
>> backups
>> (although we stopped having these since Jonathan was by and sugges= ted row
>> cache of about 1Gb, thanks Riptano!). =A0We currently have mystery= OOMs
>> which are probably caused by GC storms during compactions (althoug= h
>> usually
>> the nodes restart and compact fine, so who knows). =A0I also regul= arly watch
>> nodes go away for 30 seconds or so (logs show node goes dead, then= comes
>> back to life a few seconds later).
>>
>> I've sort of given up worrying about these, as we are in the p= rocess of
>> moving this cluster to our own machines in a colo, so I figure I s= hould
>> wait until they are moved, and see how the new machines do before = I worry
>> more about performance.
>>
>> -Anthony
>>
>> --
>> ------------------------------------------------------------------= ------
>> Anthony Molinaro =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 <anthonym@alumni.calt= ech.edu>
>
>

Any proxy with a TCP health check should be able to determine i= f the
Cassandra service is down hard. The problem for the tools that are not
cassandra protocol aware are detecting slowness or other anomalies
like TimedOut exceptions.

If you are seeing GC storms during compactions you might have rows
that are too big. When the compaction hits these memory spikes. I
lowered the compaction priority (and added more nodes) which has
helped compaction back off leaving some IO for requests.

--001636c5a57971040b048f0d6edf--