Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of gulrich@netflix.com
 designates 209.85.216.178 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAjbL_m7F-Qzhh4ycFXMQ9q3SjJRmYkq027+ACJXpRM2TjqQnQ@mail.gmail.com>
References: 
 <CAAjbL_m7F-Qzhh4ycFXMQ9q3SjJRmYkq027+ACJXpRM2TjqQnQ@mail.gmail.com>
Date: Mon, 1 Apr 2013 15:12:17 -0700
Message-ID: 
 <CAKYfx9pXWAJJoi0pbWu8D5n05JHMcRAXPo4cLYLV2=i+qyH2ZQ@mail.gmail.com>
Subject: Re: how to stop out of control compactions?
From: Gregg Ulrich <gulrich@netflix.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=047d7b6da3a8f3792704d953e980

--047d7b6da3a8f3792704d953e980
Content-Type: text/plain; charset=ISO-8859-1

You may want to set compaction threshold and not throughput.  If you set
the min threshold to something very large (100000), compactions will not
start until cassandra finds this many files to compact (which it should
not).

In the past I have used this to stop compactions on a node, and then run an
offline major compaction to get though the compaction, then set the min
threshold back.  Not everyone likes major compactions though.


  setcompactionthreshold <keyspace> <cfname> <minthreshold> <maxthreshold>
- Set the min and max
compaction thresholds for a given column family


On Mon, Apr 1, 2013 at 12:38 PM, William Oberman
<oberman@civicscience.com>wrote:

> I'll skip the prelude, but I worked myself into a bit of a jam.  I'm
> recovering now, but I want to double check if I'm thinking about things
> correct.
>
> Basically, I was in a state where a majority of my servers wanted to do
> compactions, and rather large ones.  This was impacting my site
> performance.  I tried nodetool stop COMPACTION.  I tried
> setcompactionthroughput=1.  I tried restarting servers, but they'd restart
> the compactions pretty much immediately on boot.
>
> Then I realized that:
> nodetool stop COMPACTION
> only stopped running compactions, and then the compactions would
> re-enqueue themselves rather quickly.
>
> So, right now I have:
> 1.) scripts running on N-1 servers looping on "nodetool stop COMPACTION"
> in a tight loop
> 2.) On the "Nth" server I've disabled gossip/thrift and turned up
> setcompactionthroughput to 999
> 3.) When the Nth server completes, I pick from the remaining N-1 (well,
> I'm still running the first compaction, which is going to take 12 more
> hours, but that is the plan at least).
>
> Does this make sense?  Other than the fact there was probably warning
> signs that would have prevented me from getting into this state in the
> first place? :-)
>
> will
>

--047d7b6da3a8f3792704d953e980
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div style>You may want to set compaction threshold and no=
t throughput. =A0If you set the min threshold to something very large (1000=
00), compactions will not start until cassandra finds this many files to co=
mpact (which it should not).</div>
<div style><br>In the past I have used this to stop compactions on a node, =
and then run an offline major compaction to get though the compaction, then=
 set the min threshold back. =A0Not everyone likes major compactions though=
.</div>
<div style><br></div><div style><br></div><div><br></div><div>=A0 setcompac=
tionthreshold &lt;keyspace&gt; &lt;cfname&gt; &lt;minthreshold&gt; &lt;maxt=
hreshold&gt; - Set the min and max=A0</div><div>compaction thresholds for a=
 given column family</div>
<div><br></div></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail=
_quote">On Mon, Apr 1, 2013 at 12:38 PM, William Oberman <span dir=3D"ltr">=
&lt;<a href=3D"mailto:oberman@civicscience.com" target=3D"_blank">oberman@c=
ivicscience.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">I&#39;ll skip the prelude, =
but I worked myself into a bit of a jam. =A0I&#39;m recovering now, but I w=
ant to double check if I&#39;m thinking about things correct.<div>
<br></div><div>Basically, I was in a state where a majority of my servers w=
anted to do compactions, and rather large ones. =A0This was impacting my si=
te performance. =A0I tried nodetool stop COMPACTION. =A0I tried setcompacti=
onthroughput=3D1. =A0I tried=A0restarting servers, but they&#39;d restart t=
he compactions pretty much immediately on boot.</div>


<div><br></div><div>Then I realized that:</div><div>nodetool stop COMPACTIO=
N</div><div>only stopped running compactions, and then the compactions woul=
d re-enqueue themselves rather quickly.</div><div>

<br></div><div>So, right now I have:</div><div>1.) scripts running on N-1 s=
ervers looping on &quot;nodetool stop COMPACTION&quot; in a tight loop</div=
><div>2.) On the &quot;Nth&quot; server I&#39;ve disabled gossip/thrift and=
 turned up setcompactionthroughput to 999</div>


<div>3.) When the Nth server completes, I pick from the remaining N-1 (well=
, I&#39;m still running the first compaction, which is going to take 12 mor=
e hours, but that is the plan at least).</div><div><br></div>

<div>Does this make sense? =A0Other than the fact there was probably warnin=
g signs that would have prevented me from getting into this state in the fi=
rst place? :-)</div><div><br></div><div>will</div></div>

</blockquote></div><br></div>

--047d7b6da3a8f3792704d953e980--