Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BA5FBF709 for ; Tue, 2 Apr 2013 15:14:05 +0000 (UTC) Received: (qmail 39120 invoked by uid 500); 2 Apr 2013 15:14:03 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 38995 invoked by uid 500); 2 Apr 2013 15:14:02 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 38976 invoked by uid 99); 2 Apr 2013 15:14:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Apr 2013 15:14:02 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 209.85.160.52 is neither permitted nor denied by domain of oberman@civicscience.com) Received: from [209.85.160.52] (HELO mail-pb0-f52.google.com) (209.85.160.52) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Apr 2013 15:13:57 +0000 Received: by mail-pb0-f52.google.com with SMTP id mc8so292613pbc.25 for ; Tue, 02 Apr 2013 08:13:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:x-originating-ip:in-reply-to:references :from:date:message-id:subject:to:content-type:x-gm-message-state; bh=Z0db6eO1Hyb7jdoCs4csCt45+pZ9+snGR1uIvRVZ924=; b=HBdJq/ARkOxyNQ44zszx1VolxE+sadgXeJUFfEV7kSXMl2CFW4TT6V3SYiS1duovVX MJuCrHtiMxr+/K2IyVNHEAKYvEn4gVmcNNM83FYr87Y3blZoasOGonheyLxg/3uSDjwv c7jcFxAHwZ9m3u2sjSK7nVp+y32KqN72eImy/rZ+FPeQ56SYZw686o8MTQnc/plU/pfg DHirl0ipFOEoYjrkicHif8ygAxh8twvPSeQl8KyUn73dDazlojsx9mNumYFYFpbYILKf TTkreob/ry9JjTXExAbuxWnQ1jiYCULbzUoG4ti+YD5+QdEe0lDABSl/GY306Kja2t0Q 7rdg== X-Received: by 10.67.10.238 with SMTP id ed14mr25763558pad.41.1364915617260; Tue, 02 Apr 2013 08:13:37 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.57.145 with HTTP; Tue, 2 Apr 2013 08:13:16 -0700 (PDT) X-Originating-IP: [71.206.217.47] In-Reply-To: References: <80E5BF0B-A1E5-452A-815A-803119B00A77@thelastpickle.com> From: William Oberman Date: Tue, 2 Apr 2013 11:13:16 -0400 Message-ID: Subject: Re: how to stop out of control compactions? To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=047d7b15fda183fdd604d9622e81 X-Gm-Message-State: ALoCoQkwr41iZDk6LEtYdWIM+EfNccAq7XsS/WwKjKfAGLAkazd+uu2SvYTCfPITwKEjZavER+JT X-Virus-Checked: Checked by ClamAV on apache.org --047d7b15fda183fdd604d9622e81 Content-Type: text/plain; charset=ISO-8859-1 I just tried to use this setting (I'm using 1.1.9). And it appears I can't set min > 32, as that's the max max now (using nodetool at least). Not sure if JMX would allow more access, but I don't like bypassing things I don't fully understand. I think I'll just leave my compaction killers running instead (not that killing compactions constantly isn't messing with things as well....). will On Tue, Apr 2, 2013 at 10:43 AM, William Oberman wrote: > Edward, you make a good point, and I do think am getting closer to having > to increase my cluster size (I'm around ~300GB/node now). > > In my case, I think it was neither. I had one node OOM after working on a > large compaction but it continued to run in a zombie like state (constantly > GC'ing), which I didn't have an alert on. Then I had the bad luck of a > "close token" also starting a large compaction. I have RF=3 with some of > my R/W patterns at quorum, causing that segment of my cluster to get slow > (e.g. a % of of my traffic started to slow). I was running 1.1.2 (I > haven't had to poke anything for quite some time, obviously), so I upgraded > before moving on (as I saw a lot of bug fixes to compaction issues in > release notes). But the upgrade caused even more nodes to start > compactions. Which lead to my original email... I had a cluster where 80% > of my nodes were compacting, and I really needed to boost production > traffic and couldn't seem to "tamp cassandra down" temporarily. > > Thanks for the advice everyone! > > will > > > On Tue, Apr 2, 2013 at 10:20 AM, Edward Capriolo wrote: > >> Settings do not make compactions go away. If your compactions are "out of >> control" it usually means one of these things, >> 1) you have a corrupt table that the compaction never finishes on, >> sstables count keep growing >> 2) you do not have enough hardware to handle your write load >> >> >> On Tue, Apr 2, 2013 at 7:50 AM, William Oberman > > wrote: >> >>> Thanks Gregg & Aaron. Missed that setting! >>> >>> On Tuesday, April 2, 2013, aaron morton wrote: >>> >>>> Set the min and max >>>> compaction thresholds for a given column family >>>> >>>> +1 for setting the max_compaction_threshold (as well as the min) on the >>>> a CF when you are getting behind. It can limit the size of the compactions >>>> and give things a chance to complete in a reasonable time. >>>> >>>> Cheers >>>> >>>> ----------------- >>>> Aaron Morton >>>> Freelance Cassandra Consultant >>>> New Zealand >>>> >>>> @aaronmorton >>>> http://www.thelastpickle.com >>>> >>>> On 2/04/2013, at 3:42 AM, Gregg Ulrich wrote: >>>> >>>> You may want to set compaction threshold and not throughput. If you >>>> set the min threshold to something very large (100000), compactions will >>>> not start until cassandra finds this many files to compact (which it should >>>> not). >>>> >>>> In the past I have used this to stop compactions on a node, and then >>>> run an offline major compaction to get though the compaction, then set the >>>> min threshold back. Not everyone likes major compactions though. >>>> >>>> >>>> >>>> setcompactionthreshold >>>> - Set the min and max >>>> compaction thresholds for a given column family >>>> >>>> >>>> >>>> On Mon, Apr 1, 2013 at 12:38 PM, William Oberman < >>>> oberman@civicscience.com> wrote: >>>> >>>>> I'll skip the prelude, but I worked myself into a bit of a jam. I'm >>>>> recovering now, but I want to double check if I'm thinking about things >>>>> correct. >>>>> >>>>> Basically, I was in a state where a majority of my servers wanted to >>>>> do compactions, and rather large ones. This was impacting my site >>>>> performance. I tried nodetool stop COMPACTION. I tried >>>>> setcompactionthroughput=1. I tried restarting servers, but they'd restart >>>>> the compactions pretty much immediately on boot. >>>>> >>>>> Then I realized that: >>>>> nodetool stop COMPACTION >>>>> only stopped running compactions, and then the compactions would >>>>> re-enqueue themselves rather quickly. >>>>> >>>>> So, right now I have: >>>>> 1.) scripts running on N-1 servers looping on "nodetool stop >>>>> COMPACTION" in a tight loop >>>>> 2.) On the "Nth" server I've disabled gossip/thrift and turned up >>>>> setcompactionthroughput to 999 >>>>> 3.) When the Nth server completes, I pick from the remaining N-1 >>>>> (well, I'm still running the first compaction, which is going to take 12 >>>>> more hours, but that is the plan at least). >>>>> >>>>> Does this make sense? Other than the fact there was probably warning >>>>> signs that would have prevented me from getting into this state in the >>>>> first place? :-) >>>>> >>>>> will >>>>> >>>> >>>> >>>> >>> >>> >>> --047d7b15fda183fdd604d9622e81 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I just tried to use this setting (I'm using 1.1.9). = =A0And it appears I can't set min > 32, as that's the max max no= w (using nodetool at least). =A0Not sure if JMX would allow more access, bu= t I don't like bypassing things I don't fully understand. =A0I thin= k I'll just leave my compaction killers running instead (not that killi= ng compactions constantly isn't messing with things as well....).

will


On Tue, Apr 2, 2013 at 10:43 AM, William Oberman <oberman@civicscience.com> wrote:
Edward, you make a good poi= nt, and I do think am getting closer to having to increase my cluster size = (I'm around ~300GB/node now). =A0

In my case, I think it was neither. =A0I had one node OOM af= ter working on a large compaction but it continued to run in a zombie like = state (constantly GC'ing), which I didn't have an alert on. =A0Then= I had the bad luck of a "close token" also starting a large comp= action. =A0I have RF=3D3 with some of my R/W patterns at=A0quorum, causing = that segment of my cluster to get slow (e.g. a % of of my traffic started t= o slow). =A0I was running 1.1.2 (I haven't had to poke anything for qui= te some time,=A0obviously), so I upgraded before moving on (as I saw a lot = of bug fixes to compaction issues in release notes). =A0But the upgrade cau= sed even more nodes to start compactions. =A0Which lead to my original emai= l... I had a cluster where 80% of my nodes were compacting, and I really ne= eded to boost production traffic and couldn't seem to "tamp cassan= dra down" temporarily. =A0

Thanks for the advice everyone!

will
<= br>
On Tue, Apr 2, 2013 at 10:20 AM, Edward C= apriolo <edlinuxguru@gmail.com> wrote:
Settings do not make c= ompactions go away. If your compactions are "out of control" it u= sually means one of these things,
1)=A0 you have a corrupt table that the compaction never finishe= s on, sstables count keep growing
2) you do not have enough hardware to handle your write load


On Tue= , Apr 2, 2013 at 7:50 AM, William Oberman <oberman@civicscience.com= > wrote:
Thanks Gregg & Aaron. Misse= d that setting!=A0

On Tuesday, April 2, 2013, aaron morton wrote:
Set the min and max=A0
compaction thresholds for a given column family
+1 for setting the max_compaction_threshold (as well as the min) on the = a CF when you are getting behind. It can limit the size of the compactions = and give things a chance to complete in a reasonable time.=A0

Cheers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 2/04/2013, at 3:42 AM, Gregg Ulrich <gulrich@netflix= .com> wrote:

You may want to set compaction threshold and not throughput. =A0If you= set the min threshold to something very large (100000), compactions will n= ot start until cassandra finds this many files to compact (which it should = not).

In the past I have used this to stop compactions on a node, and th= en run an offline major compaction to get though the compaction, then set t= he min threshold back. =A0Not everyone likes major compactions though.



=A0 setcompactionthreshol= d <keyspace> <cfname> <minthreshold> <maxthreshold>= - Set the min and max=A0
compaction thresholds for a given colum= n family



On Mon, Apr 1, 2013 at 12:38 PM, William Oberman = <oberman@civicscience.com> wrote:
I'll skip the prelude, = but I worked myself into a bit of a jam. =A0I'm recovering now, but I w= ant to double check if I'm thinking about things correct.

Basically, I was in a state where a majority of my servers w= anted to do compactions, and rather large ones. =A0This was impacting my si= te performance. =A0I tried nodetool stop COMPACTION. =A0I tried setcompacti= onthroughput=3D1. =A0I tried=A0restarting servers, but they'd restart t= he compactions pretty much immediately on boot.

Then I realized that:
nodetool stop COMPACTIO= N
only stopped running compactions, and then the compactions woul= d re-enqueue themselves rather quickly.

So, right now I have:
1.) scripts running on N-1 s= ervers looping on "nodetool stop COMPACTION" in a tight loop
2.) On the "Nth" server I've disabled gossip/thrift and= turned up setcompactionthroughput to 999
3.) When the Nth server completes, I pick from the remaining N-1 (well= , I'm still running the first compaction, which is going to take 12 mor= e hours, but that is the plan at least).

Does this make sense? =A0Other than the fact there was probably warnin= g signs that would have prevented me from getting into this state in the fi= rst place? :-)

will





--047d7b15fda183fdd604d9622e81--