Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of maximkr@gmail.com designates
 209.85.161.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:organization:user-agent:mime-version:to
         :subject:references:in-reply-to:content-type
         :content-transfer-encoding;
        b=jK/5CguNzLgv+rKyzypFgudgllrTdzmsEu7YDCNYNzldLg32H0+4aShMzA77zNN52G
         Cv4pnwuPFzGWo+VcM6MKl5HZIN+7/K7YKHCQohQvm+shkrqCHj9byctS6DFfzXTHP32p
         qL51DOVVJG6/VRagBs1iPjzPogMcIoA+hvN8w=
Sender: Maxim Kramarenko <maximkr@gmail.com>
Message-ID: <4C03ABB9.6050007@trackstudio.com>
Date: Mon, 31 May 2010 16:29:45 +0400
From: Maxim Kramarenko <maximkr@trackstudio.com>
Organization: TrackStudio
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.1.9) Gecko/20100330 Shredder/3.0.4
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Re: nodetool cleanup isn't cleaning up?
References: <AANLkTinaxJ1rmBXH4etBBdNJ_OcYdp5zl86F7g9e84No@mail.gmail.com>
In-Reply-To: <AANLkTinaxJ1rmBXH4etBBdNJ_OcYdp5zl86F7g9e84No@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hello!

You likely need wait for GCGraceSeconds seconds or modify this param.

http://spyced.blogspot.com/2010/02/distributed-deletes-in-cassandra.html
===
Thus, a delete operation can't just wipe out all traces of the data 
being removed immediately: if we did, and a replica did not receive the 
delete operation, when it becomes available again it will treat the 
replicas that did receive the delete as having missed a write update, 
and repair them! So, instead of wiping out data on delete, Cassandra 
replaces it with a special value called a tombstone. The tombstone can 
then be propagated to replicas that missed the initial remove request.
...
Here, we defined a constant, GCGraceSeconds, and had each node track 
tombstone age locally. Once it has aged past the constant, it can be GC'd.
===


On 31.05.2010 16:23, Ran Tavory wrote:
> I hope I understand nodetool cleanup correctly - it should clean up all
> data that does not (currently) belong to this node. If so, I think it
> might not be working correctly.
>
> Look at nodes 192.168.252.124 and 192.168.252.99 below
>
> 192.168.252.99Up         279.35 MB
> 3544607988759775661076818827414252202      |<--|
> 192.168.252.124Up         167.23 MB
> 56713727820156410577229101238628035242     |   ^
> 192.168.252.125Up         82.91 MB
>   85070591730234615865843651857942052863     v   |
> 192.168.254.57Up         366.6 MB
>   113427455640312821154458202477256070485    |   ^
> 192.168.254.58Up         88.44 MB
>   141784319550391026443072753096570088106    v   |
> 192.168.254.59Up         88.45 MB
>   170141183460469231731687303715884105727    |-->|
>
> I wanted 124 to take all the load from 99. So I issued a move command.
>
> $ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243
>
> This command tells 99 to take the space b/w
> (56713727820156410577229101238628035242, 56713727820156410577229101238628035243]
> which is basically just one item in the token space, almost nothing... I
> wanted it to be very slim (just playing around).
>
> So, next I get this:
>
> 192.168.252.124Up         803.33 MB
> 56713727820156410577229101238628035242     |<--|
> 192.168.252.99Up         352.85 MB
> 56713727820156410577229101238628035243     |   ^
> 192.168.252.125Up         134.24 MB
> 85070591730234615865843651857942052863     v   |
> 192.168.254.57Up         676.41 MB
> 113427455640312821154458202477256070485    |   ^
> 192.168.254.58Up         99.74 MB
>   141784319550391026443072753096570088106    v   |
> 192.168.254.59Up         99.94 MB
>   170141183460469231731687303715884105727    |-->|
>
> The tokens are correct, but it seems that 99 still has a lot of data.
> Why? OK, that might be b/c it didn't delete its moved data.
> So next I issued a nodetool cleanup, which should have taken care of
> that. Only that it didn't, the node 99 still has 352 MB of data. Why?
> So, you know what, I waited for 1h. Still no good, data wasn't cleaned up.
> I restarted the server. Still, data wasn't cleaned up... I issued a
> cleanup again... still no good... what's up with this node?
>
>

-- 
Best regards,
  Maxim                            mailto:maximkr@trackstudio.com

LinkedIn Profile: http://www.linkedin.com/in/maximkr
Google Talk/Jabber: maximkr@gmail.com
ICQ number: 307863079
Skype Chat: maxim.kramarenko
Yahoo! Messenger: maxim_kramarenko