Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: Pass (protection.outlook.com: domain of fold3.com designates
 66.43.16.219 as permitted sender) receiver=protection.outlook.com;
 client-ip=66.43.16.219; helo=PREXCHU02.myfamily.int;
From: Robert Wille <rwille@fold3.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Compaction no longer working properly
Thread-Topic: Compaction no longer working properly
Thread-Index: AQHQxiiG7r4INGNUTUOe31lwAODHFA==
Date: Fri, 24 Jul 2015 15:50:55 +0000
Message-ID: <80D84492-28BF-49B8-B3A0-76869380B9A0@fold3.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="Windows-1252"
Content-ID: <537E7A327B638A47BEB6CD5430A81181@ancestry.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
SN2PR0601MB766: X-MS-Exchange-Organization-RulesExecuted
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jul 2015 15:50:56.6448
 (UTC)
X-MS-Exchange-CrossTenant-Id: fcf82d21-fd42-46be-a4f5-537daf908555
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: 
 TenantId=fcf82d21-fd42-46be-a4f5-537daf908555;Ip=[66.43.16.219];Helo=[PREXCHU02.myfamily.int]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN2PR0601MB766

I have a database which has a fair amount of churn. When I need to update a=
 data structure, I create a new one, and when it is complete, I delete the =
old one. I have gc_grace_seconds=3D0, so the space for the old data structu=
res should be reclaimed on the next compaction. This has been working fine =
for months. Unfortunately, compaction has stopped doing its job, and about =
a week ago disk space started increasing steadily. Disk space usage has inc=
reased almost 50% in the last week.

I do see compaction tasks get kicked off, but they complete very quickly, g=
enerally in well less than a minute. Compaction of the bigger tables used t=
o take more than an hour. Admittedly, its been quite a while since I=92ve w=
atched compactions run, and I probably haven=92t watched them since we last=
 upgraded, so perhaps there=92s been a change in how LeveledCompaction work=
s. Our last upgrade was well more than a week ago, so an upgrade isn=92t re=
sponsible for this change in behavior. We=92re running 2.0.14 currently.

Looking at the logs, I see stuff like this:

INFO [CompactionExecutor:49180] 2015-07-24 09:31:45,960 CompactionTask.java=
 (line 296) Compacted 2 sstables to [/var/lib/cassandra/data/fold31_browse/=
node/fold31_browse-node-jb-11110,].  117,066,921 bytes to 117,346,402 (~100=
% of original) in 33,766ms =3D 3.314288MB/s.  1,444,907 total partitions me=
rged to 1,400,563.  Partition merge counts were {1:1356219, 2:44344, }

So, compaction seems to be doing something, just not very much. I=92ve chec=
ked the process that deletes stale data, and it is doing its job. Plenty of=
 deletion going on, just no space reclamation.

Any thoughts?

Thanks in advance

Robert