Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of rmenon@apigee.com designates
 209.85.223.177 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <l5aq2c$gvv$1@ger.gmane.org>
References: <l58ssf$b3u$1@ger.gmane.org>
 <CAEDUwd3m57Aj4aoZYGonnCNOH0XnsVOkrP=jhMe5NfnzTGTMmA@mail.gmail.com>
 <l5aq2c$gvv$1@ger.gmane.org>
From: Rahul Menon <rahul@apigee.com>
Date: Fri, 22 Nov 2013 10:30:35 +0530
Message-ID: 
 <CAJMN9iMStNmnLJrDaD7FA-BFxjFzdOCpqdFebLk59XZnoKWb1w@mail.gmail.com>
Subject: Re: 1.1.11: system keyspace is filling up
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a11c2190a34458204ebbce6f2

--001a11c2190a34458204ebbce6f2
Content-Type: text/plain; charset=ISO-8859-1

Oleg,

The system keyspace is not replicated it is local to the node. You should
check your logs to see if there are Timeouts from streaming hints, i
believe the default value to stream hints it 10 seconds. When i ran into
this problem i truncated hints to clear out the space and then ran a repair
so ensure that all the data was consistant across all nodes, even if there
was a failure.

-rm


On Tue, Nov 5, 2013 at 6:29 PM, Oleg Dulin <oleg.dulin@gmail.com> wrote:

>  What happens if they are not being successfully delivered ? Will they
> eventually TTL-out ?
>
>
> Also, do I need to truncate hints on every node or is it replicated ?
>
>
> Oleg
>
>
> On 2013-11-04 21:34:55 +0000, Robert Coli said:
>
>
> On Mon, Nov 4, 2013 at 11:34 AM, Oleg Dulin <oleg.dulin@gmail.com> wrote:
>
> I have a dual DC setup, 4 nodes, RF=4 in each.
>
>
> The one that is used as primary has its system keyspace fill up with 200
> gigs of data, majority of which is hints.
>
>
> Why does this happen ?
>
>
> How can I clean it up ?
>
>
> If you have this many hints, you probably have flapping / frequent network
> partition, or very overloaded nodes. If you compare the number of hints to
> the number of dropped messages, that would be informative. If you're
> hinting because you're dropping, increase capacity. If you're hinting
> because of partition, figure out why there's so much partition.
>
>
> WRT cleaning up hints, they will automatically be cleaned up eventually,
> as long as they are successfully being delivered. If you need to manually
> clean them up you can truncate system.hints keyspace.
>
>
> =Rob
>
>
>
>
>
>
> --
>
> Regards,
>
> Oleg Dulin
>
> http://www.olegdulin.com
>

--001a11c2190a34458204ebbce6f2
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:verdana,=
sans-serif;color:#000000">Oleg,=A0</div><div class=3D"gmail_default" style=
=3D"font-family:verdana,sans-serif;color:#000000"><br></div><div class=3D"g=
mail_default" style=3D"font-family:verdana,sans-serif;color:#000000">

The system keyspace is not replicated it is local to the node. You should c=
heck your logs to see if there are Timeouts from streaming hints, i believe=
 the default value to stream hints it 10 seconds. When i ran into this prob=
lem i truncated hints to clear out the space and then ran a repair so ensur=
e that all the data was consistant across all nodes, even if there was a fa=
ilure.=A0</div>

<div class=3D"gmail_default" style=3D"font-family:verdana,sans-serif;color:=
#000000"><br></div><div class=3D"gmail_default" style=3D"font-family:verdan=
a,sans-serif;color:#000000">-rm=A0</div></div><div class=3D"gmail_extra"><b=
r><br><div class=3D"gmail_quote">

On Tue, Nov 5, 2013 at 6:29 PM, Oleg Dulin <span dir=3D"ltr">&lt;<a href=3D=
"mailto:oleg.dulin@gmail.com" target=3D"_blank">oleg.dulin@gmail.com</a>&gt=
;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex">

<u></u>


<div>
<p>What happens if they are not being successfully delivered ? Will they ev=
entually TTL-out ?</p>
<p><br></p>
<p>Also, do I need to truncate hints on every node or is it replicated ?</p=
><span class=3D"HOEnZb"><font color=3D"#888888">
<p><br></p>
<p>Oleg</p></font></span><div><div class=3D"h5">
<p><br></p>
<p>On 2013-11-04 21:34:55 +0000, Robert Coli said:</p>
<p><br></p>
<p>On Mon, Nov 4, 2013 at 11:34 AM, Oleg Dulin <span>&lt;<a href=3D"mailto:=
oleg.dulin@gmail.com" target=3D"_blank"><span>oleg.dulin@gmail.com</span></=
a>&gt;</span> wrote:</p>
<p>I have a dual DC setup, 4 nodes, RF=3D4 in each.</p>
<p><br></p>
<p>The one that is used as primary has its system keyspace fill up with 200=
 gigs of data, majority of which is hints.</p>
<p><br></p>
<p>Why does this happen ?</p>
<p><br></p>
<p>How can I clean it up ?</p>
<p><br></p>
<p>If you have this many hints, you probably have flapping / frequent netwo=
rk partition, or very overloaded nodes. If you compare the number of hints =
to the number of dropped messages, that would be informative. If you&#39;re=
 hinting because you&#39;re dropping, increase capacity. If you&#39;re hint=
ing because of partition, figure out why there&#39;s so much partition.</p>


<p><br></p>
<p>WRT cleaning up hints, they will automatically be cleaned up eventually,=
 as long as they are successfully being delivered. If you need to manually =
clean them up you can truncate system.hints keyspace.</p>
<p><br></p>
<p>=3DRob</p>
<p>=A0</p>
<p><br></p>
<p><br></p>
<p><br></p>
</div></div><div class=3D"im"><p>--<span>=A0</span></p>
<p>Regards,</p>
<p>Oleg Dulin</p>
<p><a href=3D"http://www.olegdulin.com" target=3D"_blank">http://www.olegdu=
lin.com</a></p>
</div></div>
</blockquote></div><br></div>

--001a11c2190a34458204ebbce6f2--