incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From srmore <comom...@gmail.com>
Subject Re: Decommissioning a datacenter deletes the data (on decommissioned datacenter)
Date Thu, 07 Aug 2014 18:54:01 GMT
Thanks for the detailed reply Ken, this really helps. I also realized that
I wasn't doing a 'nodetool rebuild' after reading your email. I was
following the steps mentioned here
http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_decomission_dc_t.html

I do a test with nodetool rebuild and see what happens.



On Thu, Aug 7, 2014 at 1:27 PM, Ken Hancock <ken.hancock@schange.com> wrote:

> My reading is it didn't forget the schema.  It lost the data.
>

> My reading is decomissioning worked fine.  Possibly when you changed the
> replication on a keyspace to include a second data center, the data didn't
> get replicated.
>

Probably not because I could see the sstables for the keyspace from the
other datacenter created. My understanding could be wrong though.


>
> When you ADD a datacenter, you need to do a nodetool rebuild to get the
> data streamed to the new data center.  When you alter a keyspace to include
> another datacenter in its replication schema, a nodetool repair is required
> -- was this done?
> http://www.datastax.com/documentation/cql/3.0/cql/cql_using/update_ks_rf_t.html
>

I missed the 'nodetool rebuild' step that could be my issue, yes I did run
repair.


>
> When you use nodetool decomission, you're effectively deleting the
> parititioning token from the cluster.  The node being decommissioned will
> stream its data to the new owners of its original token range.  This
> streaming in no way should affect any other datacenter because you have not
> changed the tokens or data ownership for any datacenter but the one in
> which you are decomissioning a node.
>

That is what my understanding was, but when I decommission it does clear
out (removes) all the keyspaces.


>
> When you eventually decomission the last node in the datacenter, all data
> is gone as there are no tokens in that datacenter to own any data.
>
> If you had a keyspace that was only replicated within that datacenter,
> that data is gone (though you could probably add nodes back in and
> ressurect it).
>


The (now outdated) documentation [1] says that data remains on the node
even after decommissioning. So I do not understand why the data would go
away.


>
> If you had a keyspace where you changed the replication to include another
> datacenter, if that datacenter had never received the data, then it may
> have the schema but would have none of the data (other than new data that
> was written AFTER you change the replication).
>

I would expect the repair to fix this, i.e. to stream the old data to the
newly added datacenter. So, does nodetool rebuild help here ?

[1] https://wiki.apache.org/cassandra/Operations#Removing_nodes_entirely



>
>
>
>
> On Thu, Aug 7, 2014 at 2:11 PM, srmore <comomore@gmail.com> wrote:
>
>>
>>
>>
>> On Thu, Aug 7, 2014 at 12:27 PM, Robert Coli <rcoli@eventbrite.com>
>> wrote:
>>
>>> On Thu, Aug 7, 2014 at 10:04 AM, srmore <comomore@gmail.com> wrote:
>>>
>>>> Sorry for being ambiguous.  By "deletes" I mean that running
>>>> decommission I can no longer see any keyspaces owned by this node or
>>>> replicated by other nodes using the cfstats command. I am also seeing the
>>>> same behavior when I remove a single node from a cluster (without
>>>> datacenters).
>>>>
>>>
>>> I'm still not fully parsing you, but clusters should never "forget"
>>> schema as a result of decommission.
>>>
>>> Is that what you are saying is happening?
>>>
>>
>> Yes, this is what is happening.
>>
>>
>>>
>>> (In fact, even the decommissioned node itself does not forget its
>>> schema, which I personally consider a bug.)
>>>
>>
>>
>> Ok, so I am assuming this is not a normal behavior and possibly a bug  -
>> is this correct ?
>>
>>
>>>
>>> =Rob
>>>
>>>
>>
>
>
> --
> *Ken Hancock *| System Architect, Advanced Advertising
> SeaChange International
> 50 Nagog Park
> Acton, Massachusetts 01720
> ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC
> <http://www.schange.com/en-US/Company/InvestorRelations.aspx>
> Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hancock@schange.com
>  | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image:
> LinkedIn] <http://www.linkedin.com/in/kenhancock>
>
> [image: SeaChange International]
>  <http://www.schange.com/>This e-mail and any attachments may contain
> information which is SeaChange International confidential. The information
> enclosed is intended only for the addressees herein and may not be copied
> or forwarded without permission from SeaChange International.
>

Mime
View raw message