Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of mina.naguib@bloomdigital.com
 designates 209.85.216.51 as permitted sender)
Sender: Mina Naguib <mina.naguib@bloomdigital.com>
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: leveled compaction and tombstoned data
From: Mina Naguib <mina.naguib@adgear.com>
In-Reply-To: 
 <CAH0yS3JA4auTH_e6BSS4OuOk4bhTV8=sR=Dg3F3fm7NL_4MMqA@mail.gmail.com>
Date: Fri, 9 Nov 2012 09:31:09 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <A35C60BE-67CD-430C-871B-6799FC2877FB@adgear.com>
References: 
 <CAH0yS3JA4auTH_e6BSS4OuOk4bhTV8=sR=Dg3F3fm7NL_4MMqA@mail.gmail.com>
To: user@cassandra.apache.org


On 2012-11-08, at 1:12 PM, B. Todd Burruss <btoddb@gmail.com> wrote:

> we are having the problem where we have huge SSTABLEs with tombstoned =
data in them that is not being compacted soon enough (because size =
tiered compaction requires, by default, 4 like sized SSTABLEs).  this is =
using more disk space than we anticipated.
>=20
> we are very write heavy compared to reads, and we delete the data =
after N number of days (depends on the column family, but N is around 7 =
days)
>=20
> my question is would leveled compaction help to get rid of the =
tombstoned data faster than size tiered, and therefore reduce the disk =
space usage

=46rom my experience, levelled compaction makes space reclamation after =
deletes even less predictable than sized-tier.

The reason is that deletes, like all mutations, are just recorded into =
sstables.  They enter level0, and get slowly, over time, promoted =
upwards to levelN.

Depending on your *total* mutation volume VS your data set size, this =
may be quite a slow process.  This is made even worse if the size of the =
data you're deleting (say, an entire row worth several hundred =
kilobytes) is to-be-deleted by a small row-level tombstone.  If the row =
is sitting in level 4, the tombstone won't impact it until enough data =
has pushed over all existing data in level3, level2, level1, level0

Finally, to guard against the tombstone missing any data, the tombstone =
itself is not candidate for removal (I believe even after gc_grace has =
passed) unless it's reached the highest populated level in levelled =
compaction.  This means if you have 4 levels and issue a ton of deletes =
(even deletes that will never impact existing data), these tombstones =
are deadweight that cannot be purged until they hit level4.

For a write-heavy workload, I recommend you stick with sized-tier.  You =
have several options at your disposal (compaction min/max thresholds, =
gc_grace) to move things along.  If that doesn't help, I've heard of =
some fairly reputable people doing some fairly blasphemous things (major =
compactions every night).