Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
Date: Tue, 22 Dec 2015 15:05:51 +0530
Message-ID: 
 <CAPiyorWO-DFazcE6uHG9Te2AK4_FZmWZhCLR1ZY_C=vfzghNKg@mail.gmail.com>
Subject: Using TTL for data purge
From: "jaalex.tech" <jaalex.tech@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=94eb2c055098d5919b05277954cb

--94eb2c055098d5919b05277954cb
Content-Type: text/plain; charset=UTF-8

Hi,

I'm looking for suggestions/caveats on using TTL as a subsitute for a
manual data purge job.

We have few tables that hold user information - this could be guest or
registered users, and there could be between 500K to 1M records created per
day per table. Currently, these tables have a secondary indexed
updated_date column which is populated on each update. However, we have
been getting timeouts when running queries using updated_date when the
number of records are high, so i don't think this would be a reliable
option in the long term when we need to purge records that have not been
used for the last X days.

In this scenario, is it advisable to include a high enough TTL (i.e the
amount of time we want these to last, could be 3 to 6 months) when
inserting/updating records?

There could be cases where the TTL may get reset after couple of
days/weeks, when the user visits the site again.

The tables have fixed number of columns, except for one which has a
clustering key, and may have max 10 entries per  partition key.

I need to know the overhead of having so many rows with TTL hanging around
for a relatively longer duration (weeks/months), and the impacts it could
have on performance/storage. If this is not a recommended approach, what
would be an alternate design which could be used for a manual purge job,
without using secondary indices.

We are using Cassandra 2.0.x.

Thanks,
Joseph

--94eb2c055098d5919b05277954cb
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>I&#39;m looking for suggestions/cav=
eats on using TTL as a subsitute for a manual data purge job.=C2=A0</div><d=
iv><br></div><div>We have few tables that hold user information - this coul=
d be guest or registered users, and there could be between 500K to 1M recor=
ds created per day per table. Currently, these tables have a secondary inde=
xed updated_date column which is populated on each update. However, we have=
 been getting timeouts when running queries using updated_date when the num=
ber of records are high, so i don&#39;t think this would be a reliable opti=
on in the long term when we need to purge records that have not been used f=
or the last X days.=C2=A0</div><div><br></div><div>In this scenario, is it =
advisable to include a high enough TTL (i.e the amount of time we want thes=
e to last, could be 3 to 6 months) when inserting/updating records?=C2=A0</=
div><div><br></div><div>There could be cases where the TTL may get reset af=
ter couple of days/weeks, when the user visits the site again.</div><div><b=
r></div><div>The tables have fixed number of columns, except for one which =
has a clustering key, and may have max 10 entries per =C2=A0partition key.<=
/div><div><br></div><div>I need to know the overhead of having so many rows=
 with TTL hanging around for a relatively longer duration (weeks/months), a=
nd the impacts it could have on performance/storage. If this is not a recom=
mended approach, what would be an alternate design which could be used for =
a manual purge job, without using secondary indices.</div><div><br></div><d=
iv>We are using=C2=A0Cassandra 2.0.x.</div><div><br></div><div>Thanks,</div=
><div>Joseph</div><div><br></div></div>

--94eb2c055098d5919b05277954cb--