incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sylvain Lebresne <sylv...@datastax.com>
Subject Re: URGENT HELP PLEASE!
Date Fri, 25 Mar 2011 20:16:01 GMT
> Although after all the help from the Cassandra community I have a much better understanding
of why and how my situation happened, there was still one strange side effect I noticed. For
context, I store user accounts and other account information in Cassandra. When the second
node was offline and I tried to log into the site, I got an error saying invalid password.
Out of curiosity I logged into the cassandra-cli tool and looked at what columns and values
were present for my user account. My User CF seemed to have data stored from right before
I added the second node. I found that really strange assuming that Cassandra doesn't keep
any historical or versioned data? Again, once the second node was back online both servers
showed the expected more current data.

What happened is this:
You started your cluster with only one node, so at first, all data was on this.
Then you added a second node. Cassandra then moved (approximatively)
half of the data to the second node. In theory, at that
point the data that was moved to the second node could be removed from
the first node (since you had RF=1). However, Cassandra
don't do that removing part automatically for safety reasons. You'll
have to run cleanup on the first node for that to happen.
So there was stale data on the first node, that never got updated
because the first node was not responsible anymore for that data.
It was garbage that just didn't get removed. What you should have done
is run nodetool cleanup on the first node after having bootstrapped
the second one and checked everything was fine.

>
> Today I'm preparing to increase my replication factor to 2 and have been reading about
the proper way to do that. Although I've found bits and pieces, I haven't found any definitive
explanation on how to do it. Could someone please sanity check my intended approach?
>
> 1. Change the RF to 2 and restart Cassandra on both nodes
> 2. Run `nodetool repair` on both nodes, one at a time as to not halt up both servers
(will that sync data between the nodes?)
>
> In a 2 node environment and RF=2 using consistency level of ONE would still ensure data
is replicated to both servers, correct?
>
> -----Original Message-----
> From: Sylvain Lebresne [mailto:sylvain@datastax.com]
> Sent: Friday, March 25, 2011 3:01 AM
> To: user@cassandra.apache.org
> Cc: Jared Laprise
> Subject: Re: URGENT HELP PLEASE!
>
> On Fri, Mar 25, 2011 at 1:49 AM, Jared Laprise <jared@webonyx.com> wrote:
>> Hello all, I'm running 2 Cassandra 6.5 nodes and I brought down the
>> secondary node and restarted the primary node. After Cassandra came
>> back up all data has been reverted to several months ago.
>
> Out of curiosity, when you said 'brought down the secondary node', did that involved
a decomission or removeToken ? If so, I have an explanation for you.
>
> --
> Sylvain
>
>
>> I could really use some incite here, this is a production website and
>> I need to act quickly. I have a cron job that takes a snapshot every
>> night, but even with that I tried to restore a snapshot on my local
>> development environment and it was also missing a ton of data.
>>
>>
>>
>> Any help will be so appreciated.
>>
>>
>>
>>
>

Mime
View raw message