Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_EhdT58HpPziV/XaFcn+6Hw)"
Sender: beggleston@apple.com
Date: Fri, 12 May 2017 10:56:48 -0700
From: Blake Eggleston <beggleston@apple.com>
To: Stefano Ortolani <ostefano@gmail.com>, user@cassandra.apache.org
Message-id: <etPan.5915f760.33772137.e21@apple.com>
In-reply-to: <etPan.5915b868.9a6b499.e21@AirmailxGenerated.am>
References: 
 <CAKy_EwZ=eVAkDwBY12Vcw4chjjp7=0hiK4RwT2t_7SU5E_9SGA@mail.gmail.com>
 <etPan.5914b459.1a8f073d.e21@apple.com>
 <etPan.5915b868.9a6b499.e21@AirmailxGenerated.am>
Subject: Re: LCS, range tombstones, and eviction
archived-at: Fri, 12 May 2017 17:57:20 -0000

--Boundary_(ID_EhdT58HpPziV/XaFcn+6Hw)
Content-type: text/plain; charset=utf-8
Content-transfer-encoding: quoted-printable
Content-disposition: inline

The start and end points of a range tombstone are basically stored as spe=
cial purpose rows alongside the normal data in an sstable. As part of a r=
ead, they're reconciled with the data from the other sstables into a sing=
le partition, just like the other rows. The only difference is that they =
don't contain any 'real' data, and, of course, they prevent 'deleted' dat=
a from being returned in the read. It's a bit more complicated than that,=
 but that's the general idea.


On May 12, 2017 at 6:23:01 AM, Stefano Ortolani (ostefano=40gmail.com) wr=
ote:

Thanks a lot Blake, that definitely helps=21

I actually found a ticket re range tombstones and how they are accounted =
for:=C2=A0https://issues.apache.org/jira/browse/CASSANDRA-8527

I am wondering now what happens when a node receives a read request. Are =
the range tombstones read before scanning the SStables=3F More interestin=
gly, given that a single partition might be split across different levels=
, and that some range tombstones might be in L0 while all the rest of the=
 data in L1, are all the tombstones prefetched from =5Fall=5F the involve=
d SStables before doing any table scan=3F

Regards,
Stefano

On Thu, May 11, 2017 at 7:58 PM, Blake Eggleston <beggleston=40apple.com>=
 wrote:
Hi Stefano,

Based on what I understood reading the docs, if the ratio of garbage=C2=A0=

collectable tomstones exceeds the =22tombstone=5Fthreshold=22, C* should =
start=C2=A0
compacting and evicting.

If there are no other normal compaction tasks to be run, LCS will attempt=
 to compact the sstables it estimates it will be able to drop the most to=
mbstones from. It does this by estimating the number of tombstones an sst=
able has that have passed the gc grace period. Whether or not a tombstone=
 will actually be evicted is more complicated. Even if a tombstone has pa=
ssed gc grace, it can't be dropped if the data it's deleting still exists=
 in another sstable, otherwise the data would appear to return. So, a tom=
bstone won't be dropped if there is data for the same partition in other =
sstables that is older than the tombstone being evaluated for eviction.

I am quite puzzled however by what might happen when dealing with range=C2=
=A0
tombstones. In that case a single tombstone might actually stand for an=C2=
=A0
arbitrary number of normal tombstones. In other words, do range tombstone=
s=C2=A0
contribute to the =22tombstone=5Fthreshold=22=3F If so, how=3F

=46rom what I can tell, each end of the range tombstone is counted as a s=
ingle tombstone tombstone. So a range tombstone effectively contributes '=
2' to the count of tombstones for an sstable. I'm not 100% sure, but I ha=
ven't seen any sstable writing logic that tracks open tombstones and coun=
ts covered cells as tombstones. So, it's likely that the effect of range =
tombstones covering many rows are under represented in the droppable tomb=
stone estimate.

I am also a bit confused by the =22tombstone=5Fcompaction=5Finterval=22. =
If I am=C2=A0
dealing with a big partition in LCS which is receiving new records every =
day,=C2=A0
and a weekly incremental repair job continously anticompacting the data a=
nd=C2=A0
thus creating SStables, what is the likelhood of the default interval=C2=A0=

(10 days) to be actually hit=3F

It will be hit, but probably only in the repaired data. Once the data is =
marked repaired, it shouldn't be anticompacted again, and should get old =
enough to pass the compaction interval. That shouldn't be an issue though=
, because you should be running repair often enough that data is repaired=
 before it can ever get past the gc grace period. Otherwise you'll have o=
ther problems. Also, keep in mind that tombstone eviction is a part of al=
l compactions, it's just that occasionally a compaction is run specifical=
ly for that purpose. =46inally, you probably shouldn't run incremental re=
pair on data that is deleted. There is a design flaw in the incremental r=
epair used in pre-4.0 of cassandra that can cause consistency issues. It =
can also cause a *lot* of over streaming, so you might want to take a loo=
k at how much streaming your cluster is doing with full repairs, and incr=
emental repairs. It might actually be more efficient to run full repairs.=


Hope that helps,

Blake

On May 11, 2017 at 7:16:26 AM, Stefano Ortolani (ostefano=40gmail.com) wr=
ote:

Hi all,

I am trying to wrap my head around how C* evicts tombstones when using LC=
S.
Based on what I understood reading the docs, if the ratio of garbage=C2=A0=

collectable tomstones exceeds the =22tombstone=5Fthreshold=22, C* should =
start=C2=A0
compacting and evicting.

I am quite puzzled however by what might happen when dealing with range=C2=
=A0
tombstones. In that case a single tombstone might actually stand for an=C2=
=A0
arbitrary number of normal tombstones. In other words, do range tombstone=
s=C2=A0
contribute to the =22tombstone=5Fthreshold=22=3F If so, how=3F

I am also a bit confused by the =22tombstone=5Fcompaction=5Finterval=22. =
If I am=C2=A0
dealing with a big partition in LCS which is receiving new records every =
day,=C2=A0
and a weekly incremental repair job continously anticompacting the data a=
nd=C2=A0
thus creating SStables, what is the likelhood of the default interval=C2=A0=

(10 days) to be actually hit=3F

Hopefully somebody will be able to shed some lights here=21

Thanks in advance=21=C2=A0
Stefano=C2=A0


--Boundary_(ID_EhdT58HpPziV/XaFcn+6Hw)
Content-type: text/html; CHARSET=US-ASCII
Content-transfer-encoding: quoted-printable
Content-disposition: inline

<html><head><style>body=7Bfont-family:Helvetica,Arial;font-size:12px=7D</=
style></head><body style=3D=22word-wrap: break-word; -webkit-nbsp-mode: s=
pace; -webkit-line-break: after-white-space;=22><div id=3D=22bloop=5Fcust=
omfont=22 style=3D=22font-family:Helvetica,Arial;font-size:12px; color: r=
gba(0,0,0,1.0); margin: 0px; line-height: auto;=22>The start and end poin=
ts of a range tombstone are basically stored as special purpose rows alon=
gside the normal data in an sstable. As part of a read, they're reconcile=
d with the data from the other sstables into a single partition, just lik=
e the other rows. The only difference is that they don't contain any 'rea=
l' data, and, of course, they prevent 'deleted' data from being returned =
in the read. It's a bit more complicated than that, but that's the genera=
l idea.</div> <br> <div id=3D=22bloop=5Fsign=5F1494611087252628992=22 cla=
ss=3D=22bloop=5Fsign=22></div> <br><p class=3D=22airmail=5Fon=22>On May 1=
2, 2017 at 6:23:01 AM, Stefano Ortolani (<a href=3D=22mailto:ostefano=40g=
mail.com=22>ostefano=40gmail.com</a>) wrote:</p> <blockquote type=3D=22ci=
te=22 class=3D=22clean=5Fbq=22><span><div><div></div><div>


<title></title>


<div dir=3D=22ltr=22>
<div>Thanks a lot Blake, that definitely helps=21</div>
<div><br></div>
<div>I actually found a ticket re range tombstones and how they are
accounted for:&nbsp;<a href=3D=22https://issues.apache.org/jira/browse/CA=
SSANDRA-8527=22 target=3D=22=5Fblank=22>https://issues.apache.org/<wbr>ji=
ra/browse/CASSANDRA-8527</a></div>
<div><br></div>
<div>I am wondering now what happens when a node receives a read
request. Are the range tombstones read before scanning the
SStables=3F More interestingly, given that a single partition might
be split across different levels, and that some range tombstones
might be in L0 while all the rest of the data in L1, are all the
tombstones prefetched from =5Fall=5F the involved SStables before doing
any table scan=3F</div>
<div><br></div>
<div>Regards,</div>
<div>Stefano</div>
</div>
<div class=3D=22gmail=5Fextra=22><br>
<div class=3D=22gmail=5Fquote=22>On Thu, May 11, 2017 at 7:58 PM, Blake
Eggleston <span dir=3D=22ltr=22>&lt;<a href=3D=22mailto:beggleston=40appl=
e.com=22 target=3D=22=5Fblank=22>beggleston=40apple.com</a>&gt;</span> wr=
ote:<br>
<blockquote class=3D=22gmail=5Fquote=22 style=3D=22margin:0 0 0 .8ex;bord=
er-left:1px =23ccc solid;padding-left:1ex=22>
<div style=3D=22word-wrap:break-word=22>
<div id=3D=22m=5F-5124605967271343458bloop=5Fcustomfont=22 style=3D=22fon=
t-family:Helvetica,Arial;font-size:12px;color:rgba(0,0,0,1.0);margin:0px;=
line-height:auto=22>
Hi Stefano,</div>
<div id=3D=22m=5F-5124605967271343458bloop=5Fcustomfont=22 style=3D=22fon=
t-family:Helvetica,Arial;font-size:12px;color:rgba(0,0,0,1.0);margin:0px;=
line-height:auto=22>
<span class=3D=22=22><br></span>
<blockquote type=3D=22cite=22 class=3D=22m=5F-5124605967271343458clean=5F=
bq=22>
<div dir=3D=22ltr=22>
<div><span class=3D=22=22>Based on what I understood reading the docs, if=

the ratio of garbage&nbsp;</span></div>
<div><span class=3D=22=22>collectable tomstones exceeds the
=22tombstone=5Fthreshold=22, C* should start&nbsp;</span></div>
<div><span class=3D=22=22>compacting and evicting.</span></div>
</div>
</blockquote>
</div>
<div><span class=3D=22=22><br></span></div>
<div>
<div id=3D=22m=5F-5124605967271343458bloop=5Fsign=5F1494526774315814144=22=
 class=3D=22m=5F-5124605967271343458bloop=5Fsign=22></div>
If there are no other normal compaction tasks to be run, LCS will
attempt to compact the sstables it estimates it will be able to
drop the most tombstones from. It does this by estimating the
number of tombstones an sstable has that have passed the gc grace
period. Whether or not a tombstone will actually be evicted is more
complicated. Even if a tombstone has passed gc grace, it can't be
dropped if the data it's deleting still exists in another sstable,
otherwise the data would appear to return. So, a tombstone won't be
dropped if there is data for the same partition in other sstables
that is older than the tombstone being evaluated for eviction.
<div><span class=3D=22=22><br></span></div>
<div>
<blockquote type=3D=22cite=22 class=3D=22m=5F-5124605967271343458clean=5F=
bq=22>
<div dir=3D=22ltr=22>
<div><span class=3D=22=22>I am quite puzzled however by what might happen=

when dealing with range&nbsp;</span></div>
<div><span class=3D=22=22>tombstones. In that case a single tombstone
might actually stand for an&nbsp;</span></div>
<div><span class=3D=22=22>arbitrary number of normal tombstones. In other=

words, do range tombstones&nbsp;</span></div>
<div><span class=3D=22=22>contribute to the =22tombstone=5Fthreshold=22=3F=
 If so,
how=3F</span></div>
</div>
</blockquote>
</div>
<div><span class=3D=22=22><br></span></div>
<div>=46rom what I can tell, each end of the range tombstone is
counted as a single tombstone tombstone. So a range tombstone
effectively contributes '2' to the count of tombstones for an
sstable. I'm not 100% sure, but I haven't seen any sstable writing
logic that tracks open tombstones and counts covered cells as
tombstones. So, it's likely that the effect of range tombstones
covering many rows are under represented in the droppable tombstone
estimate.</div>
<div><span class=3D=22=22><br></span></div>
<div>
<blockquote type=3D=22cite=22 class=3D=22m=5F-5124605967271343458clean=5F=
bq=22>
<div dir=3D=22ltr=22>
<div><span class=3D=22=22>I am also a bit confused by the
=22tombstone=5Fcompaction=5Finterval<wbr>=22. If I am&nbsp;</span></div>
<div><span class=3D=22=22>dealing with a big partition in LCS which is
receiving new records every day,&nbsp;</span></div>
<div><span class=3D=22=22>and a weekly incremental repair job continously=

anticompacting the data and&nbsp;</span></div>
<div><span class=3D=22=22>thus creating SStables, what is the likelhood
of the default interval&nbsp;</span></div>
<div><span class=3D=22=22>(10 days) to be actually hit=3F</span></div>
</div>
</blockquote>
</div>
<div><span class=3D=22=22><br></span></div>
<div>It will be hit, but probably only in the repaired data. Once
the data is marked repaired, it shouldn't be anticompacted again,
and should get old enough to pass the compaction interval. That
shouldn't be an issue though, because you should be running repair
often enough that data is repaired before it can ever get past the
gc grace period. Otherwise you'll have other problems. Also, keep
in mind that tombstone eviction is a part of all compactions, it's
just that occasionally a compaction is run specifically for that
purpose. =46inally, you probably shouldn't run incremental repair on
data that is deleted. There is a design flaw in the incremental
repair used in pre-4.0 of cassandra that can cause consistency
issues. It can also cause a *lot* of over streaming, so you might
want to take a look at how much streaming your cluster is doing
with full repairs, and incremental repairs. It might actually be
more efficient to run full repairs.</div>
<div><br></div>
<div>Hope that helps,</div>
<div><br></div>
<div>Blake</div>
<div>
<div class=3D=22h5=22>
<div><br>
<div>
<div>
<p class=3D=22m=5F-5124605967271343458airmail=5Fon=22>On May 11, 2017 at
7:16:26 AM, Stefano Ortolani (<a href=3D=22mailto:ostefano=40gmail.com=22=
 target=3D=22=5Fblank=22>ostefano=40gmail.com</a>) wrote:</p>
<blockquote type=3D=22cite=22 class=3D=22m=5F-5124605967271343458clean=5F=
bq=22>
<div>
<div>
<div dir=3D=22ltr=22>
<div><span>Hi all,</span></div>
<div><span><br></span></div>
<div><span>I am trying to wrap my head around how C* evicts
tombstones when using LCS.</span></div>
<div><span>Based on what I understood reading the docs, if the
ratio of garbage&nbsp;</span></div>
<div><span>collectable tomstones exceeds the =22tombstone=5Fthreshold=22,=

C* should start&nbsp;</span></div>
<div><span>compacting and evicting.</span></div>
<div><span><br></span></div>
<div><span>I am quite puzzled however by what might happen when
dealing with range&nbsp;</span></div>
<div><span>tombstones. In that case a single tombstone might
actually stand for an&nbsp;</span></div>
<div><span>arbitrary number of normal tombstones. In other words,
do range tombstones&nbsp;</span></div>
<div><span>contribute to the =22tombstone=5Fthreshold=22=3F If so,
how=3F</span></div>
<div><span><br></span></div>
<div><span>I am also a bit confused by the
=22tombstone=5Fcompaction=5Finterval<wbr>=22. If I am&nbsp;</span></div>
<div><span>dealing with a big partition in LCS which is receiving
new records every day,&nbsp;</span></div>
<div><span>and a weekly incremental repair job continously
anticompacting the data and&nbsp;</span></div>
<div><span>thus creating SStables, what is the likelhood of the
default interval&nbsp;</span></div>
<div><span>(10 days) to be actually hit=3F</span></div>
<div><span><br></span></div>
<div><span>Hopefully somebody will be able to shed some lights
here=21</span></div>
<div><span><br></span></div>
<div><span>Thanks in advance=21&nbsp;</span></div>
<div><span>Stefano&nbsp;</span></div>
<div><span><br></span></div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br></div>


</div></div></span></blockquote></body></html>=

--Boundary_(ID_EhdT58HpPziV/XaFcn+6Hw)--