incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sylvain Lebresne <sylv...@yakaz.com>
Subject Re: nodetool-compact duplicated data files again and again
Date Thu, 18 Mar 2010 10:20:30 GMT
I believe this is caused by two things (and sorry if I go into too
much details):

1) there is http://wiki.apache.org/cassandra/FAQ#i_deleted_what_gives.
That is, Cassandra has to wait GCGraceSeconds before really remove physically
deleted columns. And by default, this is 10 days. For "normal" column (not the
Expiring ones of the patch), this is mitigated by the fact that only a
marker that
the column has been deleted is kept. That is, if you have 1K columns
each holding
a blob of 50Mo and you deletes them, after the first compaction the blobs are
deleted but not the columns. So you end up with you 1K columns but they now are
small. Those column will really be deleted only during a compaction that occurs
GCGraceSeconds after the deletion. But for ExpiringColumn there is 2).

2) For expired columns, the value is not deleted until the whole
column is removed,
that is, even though the column doesn't show up in a request, nothing
gets deleted
before GCGraceSeconds after the column expiration. Theoretically, what is done
for deleted columns could be done for expiring columns, that is, when
a column is
expired, the value could be removed even though the column is kept as a marker.
However this is a bit technically tricky. The natural place to do such
thing would
be when the column is serialized to disk. But the size of the
serialized column has
to be known before the actual serialization during the row indexing.
So column that
expiring in the time between the indexing and its serialization would
screw us up.
There would be way to get around that but they are not without default
and since
until now I have been able to live with that, I've moved this
'optimisation' for later.

If you yourself cannot leave with that for now, feel free to let me now.

--
Sylvain

On Wed, Mar 17, 2010 at 10:28 PM, Weijun Li <weijunli@gmail.com> wrote:
> I'm testing the ExpiringColumn patch in 0.6-beta2, inserted 26GB data with
> TTL, after columns have expired I use get_slice to verify that no columns
> can be retrieved. When I run "nodetool compact" I think all data should be
> gone. But the problem is:
>
> 1) After the first nodetool-comact, Cassandra duplicate data files to
> data-377* and then nothing happened. Total files size become 52GB. Some 0
> bytes *.Compacted files got generated.
> 2) After the second nodetool-compact, Cassandra again generated data-378*.
> Now I got 77GB data file that contains no valid columns. (See the list at
> the end)
> 3) Now I decided to run nodetool-clean and it ended up with 50GB data files
> like:
> total 53717104
> -rw-rw-r-- 1 cassandra cassandra           0 Mar 17 17:25 data-378-Compacted
> -rw-rw-r-- 1 cassandra cassandra 25563592504 Mar 17 16:25 data-378-Data.db
> -rw-rw-r-- 1 cassandra cassandra    54326245 Mar 17 16:25 data-378-Filter.db
> -rw-rw-r-- 1 cassandra cassandra  1871937928 Mar 17 16:25 data-378-Index.db
> -rw-rw-r-- 1 cassandra cassandra 25563592504 Mar 17 17:25 data-379-Data.db
> -rw-rw-r-- 1 cassandra cassandra    27163165 Mar 17 17:25 data-379-Filter.db
> -rw-rw-r-- 1 cassandra cassandra  1871937928 Mar 17 17:25 data-379-Index.db
>
> Any idea about what's going on here? I guess cleanup will remove all columns
> and don't belong this node but compact will remove all deleted columns then
> merge small files into a big one. What exactly are the differences between
> cleanup and compact?
>
> -Weijun
>
> total 80615576
> -rw-rw-r-- 1 cassandra cassandra           0 Mar 17 15:27 data-327-Compacted
> -rw-rw-r-- 1 cassandra cassandra 21013367426 Mar 16 17:43 data-327-Data.db
> -rw-rw-r-- 1 cassandra cassandra    44660005 Mar 16 17:43 data-327-Filter.db
> -rw-rw-r-- 1 cassandra cassandra  1538760208 Mar 16 17:43 data-327-Index.db
> -rw-rw-r-- 1 cassandra cassandra           0 Mar 17 15:27 data-363-Compacted
> -rw-rw-r-- 1 cassandra cassandra  2767150915 Mar 16 17:46 data-363-Data.db
> -rw-rw-r-- 1 cassandra cassandra     5890885 Mar 16 17:46 data-363-Filter.db
> -rw-rw-r-- 1 cassandra cassandra   202590655 Mar 16 17:46 data-363-Index.db
> -rw-rw-r-- 1 cassandra cassandra           0 Mar 17 15:27 data-370-Compacted
> -rw-rw-r-- 1 cassandra cassandra  1383745492 Mar 16 17:47 data-370-Data.db
> -rw-rw-r-- 1 cassandra cassandra     2947045 Mar 16 17:47 data-370-Filter.db
> -rw-rw-r-- 1 cassandra cassandra   101350867 Mar 16 17:47 data-370-Index.db
> -rw-rw-r-- 1 cassandra cassandra           0 Mar 17 15:27 data-375-Compacted
> -rw-rw-r-- 1 cassandra cassandra   345870869 Mar 16 17:50 data-375-Data.db
> -rw-rw-r-- 1 cassandra cassandra      736405 Mar 16 17:50 data-375-Filter.db
> -rw-rw-r-- 1 cassandra cassandra    25315970 Mar 16 17:50 data-375-Index.db
> -rw-rw-r-- 1 cassandra cassandra           0 Mar 17 15:27 data-376-Compacted
> -rw-rw-r-- 1 cassandra cassandra    53457802 Mar 16 18:52 data-376-Data.db
> -rw-rw-r-- 1 cassandra cassandra      113853 Mar 16 18:52 data-376-Filter.db
> -rw-rw-r-- 1 cassandra cassandra     3920228 Mar 16 18:52 data-376-Index.db
> -rw-rw-r-- 1 cassandra cassandra           0 Mar 17 16:25 data-377-Compacted
> -rw-rw-r-- 1 cassandra cassandra 25563592504 Mar 17 15:27 data-377-Data.db
> -rw-rw-r-- 1 cassandra cassandra    54327685 Mar 17 15:27 data-377-Filter.db
> -rw-rw-r-- 1 cassandra cassandra  1871937928 Mar 17 15:27 data-377-Index.db
> -rw-rw-r-- 1 cassandra cassandra 25563592504 Mar 17 16:25 data-378-Data.db
> -rw-rw-r-- 1 cassandra cassandra    54326245 Mar 17 16:25 data-378-Filter.db
> -rw-rw-r-- 1 cassandra cassandra  1871937928 Mar 17 16:25 data-378-Index.db
>

Mime
View raw message