Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Hiller, Dean" <Dean.Hiller@nrel.gov>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Tue, 28 May 2013 13:26:37 -0600
Subject: Re: data clean up problem
Thread-Topic: data clean up problem
Thread-Index: Ac5b2UYo+NWD+d7aRg2xTgUGph8FOA==
Message-ID: <CDCA6067.29811%Dean.Hiller@nrel.gov>
In-Reply-To: 
 <CAJV_UYfv-cF_-mcAWawWbdN63fwY=cULkbup0aZD8_FyLeih-g@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.3.4.130416
acceptlanguage: en-US
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Don't do any delete !=3D "need to free the disk space after retention perio=
d" which you have in both your emails.  My understanding is TTL is an expir=
y and just like tombstones will only be really deleted upon a compaction(ie=
. You do have deletes via TTL from the sound of it).  If you have TTL of 1 =
day, it does not immediately go away until the next compaction is run and t=
hen the compaction may not run on all rows???   I am not quite sure there e=
xcept the data stays in the sstable until it is compacted into a new sstabl=
e and is thrown away then as long as TTL has passed.

Dean

From: cem <cayiroglu@gmail.com<mailto:cayiroglu@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <us=
er@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Tuesday, May 28, 2013 1:17 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cas=
sandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: data clean up problem

Thanks for the answer but it is already set to 0 since I don't do any delet=
e.

Cem


On Tue, May 28, 2013 at 9:03 PM, Edward Capriolo <edlinuxguru@gmail.com<mai=
lto:edlinuxguru@gmail.com>> wrote:
You need to change the gc_grace time of the column family. It defaults to 1=
0 days. By default the tombstones will not go away for 10 days.


On Tue, May 28, 2013 at 2:46 PM, cem <cayiroglu@gmail.com<mailto:cayiroglu@=
gmail.com>> wrote:
Hi Experts,


We have general problem about cleaning up data from the disk. I need to fre=
e the disk space after retention period and the customer wants to dimension=
 the disk space base on that.

After running multiple performance tests with TTL of 1 day we saw that the =
compaction couldn't keep up with the request rate. Disks were getting full =
after 3 days. There were also a lot of sstables that are older than 1 day a=
fter 3 days.

Things that we tried:

-Change the compaction strategy to leveled. (helped a bit but not much)

-Use big sstable size (10G) with leveled compaction to have more aggressive=
 compaction.(helped a bit but not much)

-Upgrade Cassandra from 1.0 to 1.2 to use TTL histograms (didn't help at al=
l since it has key overlapping estimation algorithm that generates %100 mat=
ch. Although we don't have...)

Our column family structure is like this:

Event_data_cf: (we store event data. Event_id  is randomly generated and ea=
ch event has attributes like location=3Dlondon)

row                  data

event id          data blob

timeseries_cf: (key is the attribute that we want to index. It can be locat=
ion=3Dlondon, we didnt use secondary indexes because the indexes are dynami=
c.)

row                  data

index key       time series of event id (event1_id, event2_id=85.)

timeseries_inv_cf: (this is used for removing event by event row key. )

row                  data

event id          set of index keys

Candidate Solution: Implementing time range partitions.

Each partition will have column family set and will be managed by client.

Suppose that you want to have 7 days retention period. Then you can configu=
re the partition size as 1 day and have 7 active partitions at any time. Th=
en you can drop inactive partitions (older that 7 days). Dropping will imme=
diate remove the data from the disk. (With proper Cassandra.yaml configurat=
ion)

Storing an event:

Find the current partition p1

store to event_data to Event_data_cf_p1

store to indexes to timeseries_cff_p1

store to inverted indexes to timeseries_inv_cf_p1


A time range query with an index:

Find the all partitions belongs to that time range

Do read starting from the first partition until you reach to limit

.....

Could you please provide your comments and concerns ?

Is there any other option that we can try?

What do you think about the candidate solution?

Does anyone have the same issue? How would you solve it in another way?


Thanks in advance!

Cem