Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F285EF525 for ; Tue, 2 Apr 2013 14:43:56 +0000 (UTC) Received: (qmail 11717 invoked by uid 500); 2 Apr 2013 14:43:54 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 11650 invoked by uid 500); 2 Apr 2013 14:43:54 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 11634 invoked by uid 99); 2 Apr 2013 14:43:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Apr 2013 14:43:54 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 209.85.220.50 is neither permitted nor denied by domain of oberman@civicscience.com) Received: from [209.85.220.50] (HELO mail-pa0-f50.google.com) (209.85.220.50) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Apr 2013 14:43:49 +0000 Received: by mail-pa0-f50.google.com with SMTP id bg2so332188pad.9 for ; Tue, 02 Apr 2013 07:43:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:x-originating-ip:in-reply-to:references :from:date:message-id:subject:to:content-type:x-gm-message-state; bh=EFqe+VGWOvd4D+8eth/jR0g85pSj9c3AJKPT0So8nY8=; b=gFcab8EStvylDOO/Z4wUFvNYGjX8h4+IC2/XZ0kB4XZPm96XdhQQ+jQH60S/oyupWH 7LKJ/fO2pxoDOTu6mIl7o/9EwAk2FbgTgauHOu2FxTcLRckJY50DkJ+88GjqexIxAm1J UQ0Uo61NhU0cukcFGU1KoyMIAliI8mNCmGNCILUMurDAijM1jUikNdFHZ6/g8UxWfVDk wgNEYJtvhKy0moiUMTLWy4GJn2X60GkpEyUYW6dBHKrAGBockX/NG/wsOxSK1AjMJ4p9 0hq/keumgI9iUkyklYVdtfhTMiHSPMl53jcwSV6x7oBDGHzE9hFk3leD5LoyTkfq8aXh o4MQ== X-Received: by 10.66.123.77 with SMTP id ly13mr25340875pab.98.1364913808145; Tue, 02 Apr 2013 07:43:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.57.145 with HTTP; Tue, 2 Apr 2013 07:43:08 -0700 (PDT) X-Originating-IP: [71.206.217.47] In-Reply-To: References: <80E5BF0B-A1E5-452A-815A-803119B00A77@thelastpickle.com> From: William Oberman Date: Tue, 2 Apr 2013 10:43:08 -0400 Message-ID: Subject: Re: how to stop out of control compactions? To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=60eb69fdf43baffa8904d961c27f X-Gm-Message-State: ALoCoQnNlv9nC4ob0ZvWjF8KvNDnhwaTKJIZPB7fEzx/ptKR7Z0UmLI+ADS45+j7AeRLCxlY7pYd X-Virus-Checked: Checked by ClamAV on apache.org --60eb69fdf43baffa8904d961c27f Content-Type: text/plain; charset=ISO-8859-1 Edward, you make a good point, and I do think am getting closer to having to increase my cluster size (I'm around ~300GB/node now). In my case, I think it was neither. I had one node OOM after working on a large compaction but it continued to run in a zombie like state (constantly GC'ing), which I didn't have an alert on. Then I had the bad luck of a "close token" also starting a large compaction. I have RF=3 with some of my R/W patterns at quorum, causing that segment of my cluster to get slow (e.g. a % of of my traffic started to slow). I was running 1.1.2 (I haven't had to poke anything for quite some time, obviously), so I upgraded before moving on (as I saw a lot of bug fixes to compaction issues in release notes). But the upgrade caused even more nodes to start compactions. Which lead to my original email... I had a cluster where 80% of my nodes were compacting, and I really needed to boost production traffic and couldn't seem to "tamp cassandra down" temporarily. Thanks for the advice everyone! will On Tue, Apr 2, 2013 at 10:20 AM, Edward Capriolo wrote: > Settings do not make compactions go away. If your compactions are "out of > control" it usually means one of these things, > 1) you have a corrupt table that the compaction never finishes on, > sstables count keep growing > 2) you do not have enough hardware to handle your write load > > > On Tue, Apr 2, 2013 at 7:50 AM, William Oberman wrote: > >> Thanks Gregg & Aaron. Missed that setting! >> >> On Tuesday, April 2, 2013, aaron morton wrote: >> >>> Set the min and max >>> compaction thresholds for a given column family >>> >>> +1 for setting the max_compaction_threshold (as well as the min) on the >>> a CF when you are getting behind. It can limit the size of the compactions >>> and give things a chance to complete in a reasonable time. >>> >>> Cheers >>> >>> ----------------- >>> Aaron Morton >>> Freelance Cassandra Consultant >>> New Zealand >>> >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 2/04/2013, at 3:42 AM, Gregg Ulrich wrote: >>> >>> You may want to set compaction threshold and not throughput. If you set >>> the min threshold to something very large (100000), compactions will not >>> start until cassandra finds this many files to compact (which it should >>> not). >>> >>> In the past I have used this to stop compactions on a node, and then run >>> an offline major compaction to get though the compaction, then set the min >>> threshold back. Not everyone likes major compactions though. >>> >>> >>> >>> setcompactionthreshold >>> - Set the min and max >>> compaction thresholds for a given column family >>> >>> >>> >>> On Mon, Apr 1, 2013 at 12:38 PM, William Oberman < >>> oberman@civicscience.com> wrote: >>> >>>> I'll skip the prelude, but I worked myself into a bit of a jam. I'm >>>> recovering now, but I want to double check if I'm thinking about things >>>> correct. >>>> >>>> Basically, I was in a state where a majority of my servers wanted to do >>>> compactions, and rather large ones. This was impacting my site >>>> performance. I tried nodetool stop COMPACTION. I tried >>>> setcompactionthroughput=1. I tried restarting servers, but they'd restart >>>> the compactions pretty much immediately on boot. >>>> >>>> Then I realized that: >>>> nodetool stop COMPACTION >>>> only stopped running compactions, and then the compactions would >>>> re-enqueue themselves rather quickly. >>>> >>>> So, right now I have: >>>> 1.) scripts running on N-1 servers looping on "nodetool stop >>>> COMPACTION" in a tight loop >>>> 2.) On the "Nth" server I've disabled gossip/thrift and turned up >>>> setcompactionthroughput to 999 >>>> 3.) When the Nth server completes, I pick from the remaining N-1 (well, >>>> I'm still running the first compaction, which is going to take 12 more >>>> hours, but that is the plan at least). >>>> >>>> Does this make sense? Other than the fact there was probably warning >>>> signs that would have prevented me from getting into this state in the >>>> first place? :-) >>>> >>>> will >>>> >>> >>> >>> >> >> -- >> Will Oberman >> Civic Science, Inc. >> 6101 Penn Avenue, Fifth Floor >> Pittsburgh, PA 15206 >> (M) 412-480-7835 >> (E) oberman@civicscience.com >> > > --60eb69fdf43baffa8904d961c27f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Edward, you make a good point, and I do think am getting c= loser to having to increase my cluster size (I'm around ~300GB/node now= ). =A0

In my case, I think it was neither. =A0I had one = node OOM after working on a large compaction but it continued to run in a z= ombie like state (constantly GC'ing), which I didn't have an alert = on. =A0Then I had the bad luck of a "close token" also starting a= large compaction. =A0I have RF=3D3 with some of my R/W patterns at=A0quoru= m, causing that segment of my cluster to get slow (e.g. a % of of my traffi= c started to slow). =A0I was running 1.1.2 (I haven't had to poke anyth= ing for quite some time,=A0obviously), so I upgraded before moving on (as I= saw a lot of bug fixes to compaction issues in release notes). =A0But the = upgrade caused even more nodes to start compactions. =A0Which lead to my or= iginal email... I had a cluster where 80% of my nodes were compacting, and = I really needed to boost production traffic and couldn't seem to "= tamp cassandra down" temporarily. =A0

Thanks for the advice everyone!

will


On Tue, Apr 2, 2013 at 10:20 AM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
Settings do not make c= ompactions go away. If your compactions are "out of control" it u= sually means one of these things,
1)=A0 you have a corrupt table that the compaction never finishe= s on, sstables count keep growing
2) you do not have enough hardware to handle your write load

On Tue, Apr 2, 2013 at 7:50 AM, William Oberma= n <oberman@civicscience.com> wrote:
Thanks Gregg & Aaron. Misse= d that setting!=A0

On Tuesday, April 2, 2013, aaron morton wrote:
Set the min and max=A0
compaction thresholds for a given column family
+1 for setting the max_compaction_threshold (as well as the min) on the = a CF when you are getting behind. It can limit the size of the compactions = and give things a chance to complete in a reasonable time.=A0

Cheers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 2/04/2013, at 3:42 AM, Gregg Ulrich <gulrich@netflix= .com> wrote:

You may want to set compaction threshold and not throughput. =A0If you= set the min threshold to something very large (100000), compactions will n= ot start until cassandra finds this many files to compact (which it should = not).

In the past I have used this to stop compactions on a node, and th= en run an offline major compaction to get though the compaction, then set t= he min threshold back. =A0Not everyone likes major compactions though.



=A0 setcompactionthreshol= d <keyspace> <cfname> <minthreshold> <maxthreshold>= - Set the min and max=A0
compaction thresholds for a given colum= n family



On Mon, Apr 1, 2013 at 12:38 PM, William Oberman = <oberman@civicscience.com> wrote:
I'll skip the prelude, = but I worked myself into a bit of a jam. =A0I'm recovering now, but I w= ant to double check if I'm thinking about things correct.

Basically, I was in a state where a majority of my servers w= anted to do compactions, and rather large ones. =A0This was impacting my si= te performance. =A0I tried nodetool stop COMPACTION. =A0I tried setcompacti= onthroughput=3D1. =A0I tried=A0restarting servers, but they'd restart t= he compactions pretty much immediately on boot.

Then I realized that:
nodetool stop COMPACTIO= N
only stopped running compactions, and then the compactions woul= d re-enqueue themselves rather quickly.

So, right now I have:
1.) scripts running on N-1 s= ervers looping on "nodetool stop COMPACTION" in a tight loop
2.) On the "Nth" server I've disabled gossip/thrift and= turned up setcompactionthroughput to 999
3.) When the Nth server completes, I pick from the remaining N-1 (well= , I'm still running the first compaction, which is going to take 12 mor= e hours, but that is the plan at least).

Does this make sense? =A0Other than the fact there was probably warnin= g signs that would have prevented me from getting into this state in the fi= rst place? :-)

will




--
Will Oberman
Civic Science, Inc.
6101 Penn Avenue, = Fifth Floor
Pittsburgh, PA 15206
(M) 412-480-7835
(E) oberman@c= ivicscience.com




<= /div> --60eb69fdf43baffa8904d961c27f--