incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Tavory <ran...@gmail.com>
Subject Re: nodetool cleanup isn't cleaning up?
Date Mon, 31 May 2010 13:00:58 GMT
Do you think it's the tombstones that take up the disk space?
Shouldn't the tombstones be moved along with the data?

On Mon, May 31, 2010 at 3:29 PM, Maxim Kramarenko
<maximkr@trackstudio.com>wrote:

> Hello!
>
> You likely need wait for GCGraceSeconds seconds or modify this param.
>
> http://spyced.blogspot.com/2010/02/distributed-deletes-in-cassandra.html
> ===
> Thus, a delete operation can't just wipe out all traces of the data being
> removed immediately: if we did, and a replica did not receive the delete
> operation, when it becomes available again it will treat the replicas that
> did receive the delete as having missed a write update, and repair them! So,
> instead of wiping out data on delete, Cassandra replaces it with a special
> value called a tombstone. The tombstone can then be propagated to replicas
> that missed the initial remove request.
> ...
> Here, we defined a constant, GCGraceSeconds, and had each node track
> tombstone age locally. Once it has aged past the constant, it can be GC'd.
> ===
>
>
>
> On 31.05.2010 16:23, Ran Tavory wrote:
>
>> I hope I understand nodetool cleanup correctly - it should clean up all
>> data that does not (currently) belong to this node. If so, I think it
>> might not be working correctly.
>>
>> Look at nodes 192.168.252.124 and 192.168.252.99 below
>>
>> 192.168.252.99Up         279.35 MB
>> 3544607988759775661076818827414252202      |<--|
>> 192.168.252.124Up         167.23 MB
>> 56713727820156410577229101238628035242     |   ^
>> 192.168.252.125Up         82.91 MB
>>  85070591730234615865843651857942052863     v   |
>> 192.168.254.57Up         366.6 MB
>>  113427455640312821154458202477256070485    |   ^
>> 192.168.254.58Up         88.44 MB
>>  141784319550391026443072753096570088106    v   |
>> 192.168.254.59Up         88.45 MB
>>  170141183460469231731687303715884105727    |-->|
>>
>> I wanted 124 to take all the load from 99. So I issued a move command.
>>
>> $ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243
>>
>> This command tells 99 to take the space b/w
>> (56713727820156410577229101238628035242,
>> 56713727820156410577229101238628035243]
>> which is basically just one item in the token space, almost nothing... I
>> wanted it to be very slim (just playing around).
>>
>> So, next I get this:
>>
>> 192.168.252.124Up         803.33 MB
>> 56713727820156410577229101238628035242     |<--|
>> 192.168.252.99Up         352.85 MB
>> 56713727820156410577229101238628035243     |   ^
>> 192.168.252.125Up         134.24 MB
>> 85070591730234615865843651857942052863     v   |
>> 192.168.254.57Up         676.41 MB
>> 113427455640312821154458202477256070485    |   ^
>> 192.168.254.58Up         99.74 MB
>>  141784319550391026443072753096570088106    v   |
>> 192.168.254.59Up         99.94 MB
>>  170141183460469231731687303715884105727    |-->|
>>
>> The tokens are correct, but it seems that 99 still has a lot of data.
>> Why? OK, that might be b/c it didn't delete its moved data.
>> So next I issued a nodetool cleanup, which should have taken care of
>> that. Only that it didn't, the node 99 still has 352 MB of data. Why?
>> So, you know what, I waited for 1h. Still no good, data wasn't cleaned up.
>> I restarted the server. Still, data wasn't cleaned up... I issued a
>> cleanup again... still no good... what's up with this node?
>>
>>
>>
> --
> Best regards,
>  Maxim                            mailto:maximkr@trackstudio.com
>
> LinkedIn Profile: http://www.linkedin.com/in/maximkr
> Google Talk/Jabber: maximkr@gmail.com
> ICQ number: 307863079
> Skype Chat: maxim.kramarenko
> Yahoo! Messenger: maxim_kramarenko
>

Mime
View raw message