Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of ben@instaclustr.com designates
 209.85.192.177 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CACV_4ivFJ4i+fRPPmJNWRUwW6zqdi6cDURxs9z4Hq1YVOs-gNA@mail.gmail.com>
References: 
 <CACV_4ivNCxfz8DaqPZupe7t=RqiFftAMCf6u8m=ATFkcD9p4hg@mail.gmail.com>
	<54792650.219886.1426544924926.JavaMail.yahoo@mail.yahoo.com>
	<CACV_4it5YDUrqouUFbTv=muT+U-xXEiOH6xasuYY+R6OrBDAZA@mail.gmail.com>
	<CAA=6J0-RJVCRrH7tCLNBkF+3yeffebFgko6bnEWBXV1r5RzMyg@mail.gmail.com>
	<CACV_4ivkH17n7WXvYnoheJDnvRZ4Zam55kr-+sQvX=d4YK_WCA@mail.gmail.com>
	<CAA=6J08+K3R65SmBibiovE2LSqHJ2VFbE7D6sttg15_Gw6by1g@mail.gmail.com>
	<CACV_4ivFJ4i+fRPPmJNWRUwW6zqdi6cDURxs9z4Hq1YVOs-gNA@mail.gmail.com>
Date: Fri, 20 Mar 2015 09:52:02 -0700
Message-ID: 
 <CAA=6J0-Y1Yp4CZsa7g=4fB1tb_bYBxTKva4H2cytxM62rUG4pQ@mail.gmail.com>
Subject: Re: Deleted snapshot files filling up /var/lib/cassandra
From: Ben Bromhead <ben@instaclustr.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a11362a8ab290f90511bb229c

--001a11362a8ab290f90511bb229c
Content-Type: text/plain; charset=UTF-8

Sorry for the late reply.

To immediately solve the problem you can restart Cassandra and all the open
file descriptors to the deleted snapshots should disappear.

As for why it happened I would first address the disk space issue and see
if the snapshot errors + open file descriptors issue still occurs (I am
unclear as to whether you got the snapshot exception after the disk filled
up or before), if you still have issues with repair not letting go of
snapshotted files even with free disk space I would look to raise a ticket
in Jira.

On 17 March 2015 at 12:46, David Wahler <dwahler@indeed.com> wrote:

> On Mon, Mar 16, 2015 at 6:51 PM, Ben Bromhead <ben@instaclustr.com> wrote:
> > If you are running a sequential repair (or have previously run a
> sequential
> > repair that is still running) Cassandra will still have the file
> descriptors
> > open for files in the snapshot it is using for the repair operation.
>
> Yeah, that aligns with my understanding of how the repair process
> works. But the cluster has no repair sessions active (I think; when I
> run "nodetool tpstats", the AntiEntropyStage and AntiEntropySessions
> values are zero on all nodes) and the space still hasn't been freed.
>


-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | (650) 284 9692

--001a11362a8ab290f90511bb229c
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Sorry for the late reply.=C2=A0</div><div><br></div><=
div>To immediately solve the problem you can restart Cassandra and all the =
open file descriptors to the deleted snapshots should disappear.=C2=A0</div=
><div><br></div><div>As for why it happened I would first address the disk =
space issue and see if the snapshot errors + open file descriptors issue st=
ill occurs (I am unclear as to whether you got the snapshot exception after=
 the disk filled up or before), if you still have issues with repair not le=
tting go of snapshotted files even with free disk space I would look to rai=
se a ticket in Jira.</div></div><div class=3D"gmail_extra"><br><div class=
=3D"gmail_quote">On 17 March 2015 at 12:46, David Wahler <span dir=3D"ltr">=
&lt;<a href=3D"mailto:dwahler@indeed.com" target=3D"_blank">dwahler@indeed.=
com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=3D"=
">On Mon, Mar 16, 2015 at 6:51 PM, Ben Bromhead &lt;<a href=3D"mailto:ben@i=
nstaclustr.com">ben@instaclustr.com</a>&gt; wrote:<br>
&gt; If you are running a sequential repair (or have previously run a seque=
ntial<br>
&gt; repair that is still running) Cassandra will still have the file descr=
iptors<br>
&gt; open for files in the snapshot it is using for the repair operation.<b=
r>
<br>
</span>Yeah, that aligns with my understanding of how the repair process<br=
>
works. But the cluster has no repair sessions active (I think; when I<br>
run &quot;nodetool tpstats&quot;, the AntiEntropyStage and AntiEntropySessi=
ons<br>
values are zero on all nodes) and the space still hasn&#39;t been freed.<br=
>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr">


<p>Ben Bromhead</p>
<p>Instaclustr |=C2=A0<a href=3D"https://www.instaclustr.com/" target=3D"_b=
lank"><span>www.instaclustr.com</span></a>=C2=A0|=C2=A0<a href=3D"http://tw=
itter.com/instaclustr" target=3D"_blank"><span>@instaclustr</span></a>=C2=
=A0| (650) 284 9692<br></p></div></div></div></div>
</div>

--001a11362a8ab290f90511bb229c--