Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of dsjas297@gmail.com designates
 74.125.82.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAEDUwd1+Yj=T-zPvzg8+otKrf_JY-sESg7zMf6QtUrPf2FAaXw@mail.gmail.com>
References: 
 <CA+r1nRRsDxyd6ngqbi0VU=Tupxe9jGenxcguA5PN+4Ljcr+TCg@mail.gmail.com>
	<CAEDUwd1+Yj=T-zPvzg8+otKrf_JY-sESg7zMf6QtUrPf2FAaXw@mail.gmail.com>
Date: Fri, 25 Oct 2013 13:20:34 -0700
Message-ID: 
 <CA+r1nRSK9ZFoOPJrNparpTi5mu5EfPFSU9nKTY408yNmF=80vQ@mail.gmail.com>
Subject: Re: Cassandra SSTable deletion/load reporting question
From: Jasdeep Hundal <dsjas297@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=047d7bf10aa894788e04e9967b03

--047d7bf10aa894788e04e9967b03
Content-Type: text/plain; charset=ISO-8859-1

Thanks Rob.

Will checkout the tool you linked to. In our case it's definitely not the
tombstones hanging around since we write entire rows at once and the amount
of data in a row is far, far greater than the space a tombstone takes.

Jasdeep


On Fri, Oct 25, 2013 at 1:14 PM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Fri, Oct 25, 2013 at 1:10 PM, Jasdeep Hundal <dsjas297@gmail.com>wrote:
>
>>
>> After performing a large set of deletes on our cluster, a few hundred
>> gigabytes work (essentially cleaning out nearly all old data), we noticed
>> that nodetool reported about the same load as before.
>>
>
> Tombstones are purgeable only after gc_grace_seconds has elapsed, and only
> if all SSTables which contain fragments of that row are involved in the
> compaction.
>
>
>> With my understanding, running a repair should have triggered compactions
>> between SSTable files and reference counting on the subsequent restart of
>> cassandra on a node should have cleared the old files, but this did not
>> appear to happen. The load did not start going down until we were writing
>> to the cluster again.
>>
>
> Repair is unrelated to minor compaction, except in that it creates new
> SSTables via streaming, which may trigger minor compaction.
>
>
>> I suspect that there are a few values hanging around in the old tables so
>> the references stayed intact, can anyone confirm this?
>>
>
> Stop suspecting and measure with checksstablegarbage :
> https://github.com/cloudian/support-tools
>
>
>> What's a bit more important for me is being able to accurately report the
>> size of the "active" data set, since nodetool doesn't seem to be useful for
>> this. I use counters for reporting some of this, but is there a single
>> source of truth for this, especially since counters do occasionally miss
>> updates.
>>
>
> In very new versions of Cassandra, there is tracking of and metrics
> available for what percentage of data in a SSTable is expired.
>
> =Rob
>
>

--047d7bf10aa894788e04e9967b03
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks Rob.<br><br>Will checkout the tool you linked to. I=
n our case it&#39;s definitely not the tombstones hanging around since we w=
rite entire rows at once and the amount of data in a row is far, far greate=
r than the space a tombstone takes.<br>
<br>Jasdeep</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quo=
te">On Fri, Oct 25, 2013 at 1:14 PM, Robert Coli <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:rcoli@eventbrite.com" target=3D"_blank">rcoli@eventbrite.com<=
/a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div class=3D"im">On Fri, O=
ct 25, 2013 at 1:10 PM, Jasdeep Hundal <span dir=3D"ltr">&lt;<a href=3D"mai=
lto:dsjas297@gmail.com" target=3D"_blank">dsjas297@gmail.com</a>&gt;</span>=
 wrote:<br>
</div><div class=3D"gmail_extra"><div class=3D"gmail_quote"><div class=3D"i=
m">
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex"><div dir=3D"ltr"><br>After performing a large set of delet=
es on our cluster, a few hundred gigabytes work (essentially cleaning out n=
early all old data), we noticed that nodetool reported about the same load =
as before.<br>

</div></blockquote><div><br></div></div><div>Tombstones are purgeable only =
after gc_grace_seconds has elapsed, and only if all SSTables which contain =
fragments of that row are involved in the compaction.</div><div class=3D"im=
">
<div>=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px=
 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex">
<div dir=3D"ltr">With my understanding, running a repair should have trigge=
red compactions between SSTable files and reference counting on the subsequ=
ent restart of cassandra on a node should have cleared the old files, but t=
his did not appear to happen. The load did not start going down until we we=
re writing to the cluster again.</div>

</blockquote><div><br></div></div><div>Repair is unrelated to minor compact=
ion, except in that it creates new SSTables via streaming, which may trigge=
r minor compaction.</div><div class=3D"im"><div>=A0<br></div><blockquote cl=
ass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px=
;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1e=
x">

<div dir=3D"ltr"><div>I suspect that there are a few values hanging around =
in the old tables so the references stayed intact, can anyone confirm this?=
<br></div></div></blockquote><div><br></div></div><div>Stop suspecting and =
measure with checksstablegarbage :=A0<a href=3D"https://github.com/cloudian=
/support-tools" target=3D"_blank">https://github.com/cloudian/support-tools=
</a></div>
<div class=3D"im">
<div>=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px=
 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex"><div dir=3D"ltr"><div>What&#39;s a bit more =
important for me is being able to accurately report the size of the &quot;a=
ctive&quot; data set, since nodetool doesn&#39;t seem to be useful for this=
. I use counters for reporting some of this, but is there a single source o=
f truth for this, especially since counters do occasionally miss updates.<b=
r>

</div></div></blockquote><div><br></div></div><div>In very new versions of =
Cassandra, there is tracking of and metrics available for what percentage o=
f data in a SSTable is expired.</div><div><br></div><div>=3DRob</div><div>
=A0</div>
</div></div></div>
</blockquote></div><br></div>

--047d7bf10aa894788e04e9967b03--