cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kurt greaves <k...@instaclustr.com>
Subject Re: Deleted data comes back on node decommission
Date Fri, 15 Dec 2017 20:47:32 GMT
X==5. I was meant to fill that in...

On 16 Dec. 2017 07:46, "kurt greaves" <kurt@instaclustr.com> wrote:

> Yep, if you don't run cleanup on all nodes (except new node) after step x,
> when you decommissioned node 4 and 5 later on, their tokens will be
> reclaimed by the previous owner. Suddenly the data in those SSTables is now
> live again because the token ownership has changed and any data in those
> SStables will be returned.
>
> Remember new nodes only add tokens to the ring, they don't affect other
> nodes tokens, so if you remove those tokens everything goes back to how it
> was before those nodes were added.
>
> Adding a maker would be incredibly complicated. Plugs not really fit the
> design of Cassandra. Here it's probably much easier to just follow
> recommended procedure when adding and removing nodes.
>
> On 16 Dec. 2017 01:37, "Python_Max" <python.max@gmail.com> wrote:
>
> Hello, Jeff.
>
>
> Using your hint I was able to reproduce my situation on 5 VMs.
> Simplified steps are:
> 1) set up 3-node cluster
> 2) create keyspace with RF=3 and table with gc_grace_seconds=60,
> compaction_interval=10 and unchecked_tombstone_compaction=true (to force
> compaction later)
> 3) insert 10..20 records with different partition and clustering keys
> (consistency 'all')
> 4) 'nodetool flush' on all 3 nodes
> 5) add 4th node, add 5th node
> 6) using 'nodetool getendpoints' find key that moved to both 4th and 5th
> node
> 7) delete that record from table (consistency 'all')
> 8) 'nodetool flush' on all 5 nodes, wait gc_grace_seconds, 'nodetool
> compact' on nodes which responsible for that key, check that key and
> tombstone gone using sstabledump
> 9) decommission 5th node, decommission 4th node
> 10) select data from table where key=key (consistency quorum)
>
> And the row is here.
>
> It sounds like bug in cassandra but since it is documented here
> https://docs.datastax.com/en/cassandra/3.0/cassandra/operati
> ons/opsAddNodeToCluster.html I suppose this counts as feature. It would
> be better when data which stays in sstable after new node added would have
> some marker and never returned as result to select query.
>
> Thank you very much, Jeff, for pointing me in right direction.
>
>
> On 13.12.17 18:43, Jeff Jirsa wrote:
>
>> Did you run cleanup before you shrank the cluster?
>>
>>
> --
>
> Best Regards,
> Python_Max.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
>

Mime
View raw message