incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From srmore <comom...@gmail.com>
Subject Re: Decommissioning a datacenter deletes the data (on decommissioned datacenter)
Date Thu, 07 Aug 2014 19:29:02 GMT
I tried using 'nodetool rebuild' after I add the datacenters,date same
outcome,  and after I decommission my keyspaces are getting wiped out, I
don't understand this.


On Thu, Aug 7, 2014 at 1:54 PM, srmore <comomore@gmail.com> wrote:

>
> Thanks for the detailed reply Ken, this really helps. I also realized that
> I wasn't doing a 'nodetool rebuild' after reading your email. I was
> following the steps mentioned here
> http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_decomission_dc_t.html
>
> I do a test with nodetool rebuild and see what happens.
>
>
>
> On Thu, Aug 7, 2014 at 1:27 PM, Ken Hancock <ken.hancock@schange.com>
> wrote:
>
>> My reading is it didn't forget the schema.  It lost the data.
>>
>
>> My reading is decomissioning worked fine.  Possibly when you changed the
>> replication on a keyspace to include a second data center, the data didn't
>> get replicated.
>>
>
> Probably not because I could see the sstables for the keyspace from the
> other datacenter created. My understanding could be wrong though.
>
>
>>
>> When you ADD a datacenter, you need to do a nodetool rebuild to get the
>> data streamed to the new data center.  When you alter a keyspace to include
>> another datacenter in its replication schema, a nodetool repair is required
>> -- was this done?
>> http://www.datastax.com/documentation/cql/3.0/cql/cql_using/update_ks_rf_t.html
>>
>
> I missed the 'nodetool rebuild' step that could be my issue, yes I did run
> repair.
>
>
>>
>> When you use nodetool decomission, you're effectively deleting the
>> parititioning token from the cluster.  The node being decommissioned will
>> stream its data to the new owners of its original token range.  This
>> streaming in no way should affect any other datacenter because you have not
>> changed the tokens or data ownership for any datacenter but the one in
>> which you are decomissioning a node.
>>
>
> That is what my understanding was, but when I decommission it does clear
> out (removes) all the keyspaces.
>
>
>>
>> When you eventually decomission the last node in the datacenter, all data
>> is gone as there are no tokens in that datacenter to own any data.
>>
>> If you had a keyspace that was only replicated within that datacenter,
>> that data is gone (though you could probably add nodes back in and
>> ressurect it).
>>
>
>
> The (now outdated) documentation [1] says that data remains on the node
> even after decommissioning. So I do not understand why the data would go
> away.
>
>
>>
>> If you had a keyspace where you changed the replication to include
>> another datacenter, if that datacenter had never received the data, then it
>> may have the schema but would have none of the data (other than new data
>> that was written AFTER you change the replication).
>>
>
> I would expect the repair to fix this, i.e. to stream the old data to the
> newly added datacenter. So, does nodetool rebuild help here ?
>
> [1] https://wiki.apache.org/cassandra/Operations#Removing_nodes_entirely
>
>
>
>>
>>
>>
>>
>> On Thu, Aug 7, 2014 at 2:11 PM, srmore <comomore@gmail.com> wrote:
>>
>>>
>>>
>>>
>>> On Thu, Aug 7, 2014 at 12:27 PM, Robert Coli <rcoli@eventbrite.com>
>>> wrote:
>>>
>>>> On Thu, Aug 7, 2014 at 10:04 AM, srmore <comomore@gmail.com> wrote:
>>>>
>>>>> Sorry for being ambiguous.  By "deletes" I mean that running
>>>>> decommission I can no longer see any keyspaces owned by this node or
>>>>> replicated by other nodes using the cfstats command. I am also seeing
the
>>>>> same behavior when I remove a single node from a cluster (without
>>>>> datacenters).
>>>>>
>>>>
>>>> I'm still not fully parsing you, but clusters should never "forget"
>>>> schema as a result of decommission.
>>>>
>>>> Is that what you are saying is happening?
>>>>
>>>
>>> Yes, this is what is happening.
>>>
>>>
>>>>
>>>> (In fact, even the decommissioned node itself does not forget its
>>>> schema, which I personally consider a bug.)
>>>>
>>>
>>>
>>> Ok, so I am assuming this is not a normal behavior and possibly a bug  -
>>> is this correct ?
>>>
>>>
>>>>
>>>> =Rob
>>>>
>>>>
>>>
>>
>>
>> --
>>  *Ken Hancock *| System Architect, Advanced Advertising
>> SeaChange International
>> 50 Nagog Park
>> Acton, Massachusetts 01720
>> ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC
>> <http://www.schange.com/en-US/Company/InvestorRelations.aspx>
>> Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hancock@schange.com
>>  | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image:
>> LinkedIn] <http://www.linkedin.com/in/kenhancock>
>>
>> [image: SeaChange International]
>>  <http://www.schange.com/>This e-mail and any attachments may contain
>> information which is SeaChange International confidential. The information
>> enclosed is intended only for the addressees herein and may not be copied
>> or forwarded without permission from SeaChange International.
>>
>
>

Mime
View raw message