cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim Kramarenko <maxi...@trackstudio.com>
Subject Re: nodetool cleanup isn't cleaning up?
Date Mon, 31 May 2010 12:29:45 GMT
Hello!

You likely need wait for GCGraceSeconds seconds or modify this param.

http://spyced.blogspot.com/2010/02/distributed-deletes-in-cassandra.html
===
Thus, a delete operation can't just wipe out all traces of the data 
being removed immediately: if we did, and a replica did not receive the 
delete operation, when it becomes available again it will treat the 
replicas that did receive the delete as having missed a write update, 
and repair them! So, instead of wiping out data on delete, Cassandra 
replaces it with a special value called a tombstone. The tombstone can 
then be propagated to replicas that missed the initial remove request.
...
Here, we defined a constant, GCGraceSeconds, and had each node track 
tombstone age locally. Once it has aged past the constant, it can be GC'd.
===


On 31.05.2010 16:23, Ran Tavory wrote:
> I hope I understand nodetool cleanup correctly - it should clean up all
> data that does not (currently) belong to this node. If so, I think it
> might not be working correctly.
>
> Look at nodes 192.168.252.124 and 192.168.252.99 below
>
> 192.168.252.99Up         279.35 MB
> 3544607988759775661076818827414252202      |<--|
> 192.168.252.124Up         167.23 MB
> 56713727820156410577229101238628035242     |   ^
> 192.168.252.125Up         82.91 MB
>   85070591730234615865843651857942052863     v   |
> 192.168.254.57Up         366.6 MB
>   113427455640312821154458202477256070485    |   ^
> 192.168.254.58Up         88.44 MB
>   141784319550391026443072753096570088106    v   |
> 192.168.254.59Up         88.45 MB
>   170141183460469231731687303715884105727    |-->|
>
> I wanted 124 to take all the load from 99. So I issued a move command.
>
> $ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243
>
> This command tells 99 to take the space b/w
> (56713727820156410577229101238628035242, 56713727820156410577229101238628035243]
> which is basically just one item in the token space, almost nothing... I
> wanted it to be very slim (just playing around).
>
> So, next I get this:
>
> 192.168.252.124Up         803.33 MB
> 56713727820156410577229101238628035242     |<--|
> 192.168.252.99Up         352.85 MB
> 56713727820156410577229101238628035243     |   ^
> 192.168.252.125Up         134.24 MB
> 85070591730234615865843651857942052863     v   |
> 192.168.254.57Up         676.41 MB
> 113427455640312821154458202477256070485    |   ^
> 192.168.254.58Up         99.74 MB
>   141784319550391026443072753096570088106    v   |
> 192.168.254.59Up         99.94 MB
>   170141183460469231731687303715884105727    |-->|
>
> The tokens are correct, but it seems that 99 still has a lot of data.
> Why? OK, that might be b/c it didn't delete its moved data.
> So next I issued a nodetool cleanup, which should have taken care of
> that. Only that it didn't, the node 99 still has 352 MB of data. Why?
> So, you know what, I waited for 1h. Still no good, data wasn't cleaned up.
> I restarted the server. Still, data wasn't cleaned up... I issued a
> cleanup again... still no good... what's up with this node?
>
>

-- 
Best regards,
  Maxim                            mailto:maximkr@trackstudio.com

LinkedIn Profile: http://www.linkedin.com/in/maximkr
Google Talk/Jabber: maximkr@gmail.com
ICQ number: 307863079
Skype Chat: maxim.kramarenko
Yahoo! Messenger: maxim_kramarenko

Mime
View raw message