incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Coverston <ben.covers...@datastax.com>
Subject Re: URGENT HELP PLEASE!
Date Fri, 25 Mar 2011 03:58:31 GMT
Hi Jared,

Sounds like you have two nodes in the cluster. What is your replication 
factor set to? 1? 2?

Have you ever run repair? What consistency level do you use for reads 
and writes?

 From the way you are speaking it sounds like you are sending all of 
your traffic to a single node (primary, secondary).

If you never repair the ranges, and you read the data infrequently (or 
only with range slices) I can guess that your data never got replicated 
to your secondary node.

Ben

On 3/24/11 9:44 PM, Jared Laprise wrote:
> Thanks for the responses. I got everything working again, and have some ideas on why
but am not completely sure.
>
> How I got it working again was simply bring the second node back online. I was under
the assumption that all data is replicated between nodes (eventually). Am I incorrect? It
would seem that each node stores different data and delegates the read request to whichever
node holds the data. Although I've spent a lot of time with Cassandra in a single node environment
I think I may be lacking a bit of understanding on how Cassandra behaves in a clustered environment.
>
>
> -----Original Message-----
> From: Jonathan Ellis [mailto:jbellis@gmail.com]
> Sent: Thursday, March 24, 2011 8:22 PM
> To: user@cassandra.apache.org
> Cc: aaron morton
> Subject: Re: URGENT HELP PLEASE!
>
> Right, Cassandra doesn't keep old versions around so to see an old version you have to
have uncompacted data and whack the new data -- either by blowing away sstables or not replaying
the commitlog.
>
> Snapshots flush before creating their hard links, which rules out any commitlog problems.
>
> If you ran out of disk space you wouldn't get past the commitlog append, so you'd never
get new data in at all after that.
>
> Sounds like an environmental problem, not Cassandra specific.
>
> On Thu, Mar 24, 2011 at 9:10 PM, aaron morton<aaron@thelastpickle.com>  wrote:
>> Was there anything in the server logs during startup ?
>> I've not heard of this happening before and it's hard think of how /
>> why cassandra could revert it's data. Other than something external
>> playing with the files on disk Aaron On 25 Mar 2011, at 13:49, Jared
>> Laprise wrote:
>>
>> Hello all, I'm running 2 Cassandra 6.5 nodes and I brought down the
>> secondary node and restarted the primary node. After Cassandra came
>> back up all data has been reverted to several months ago.
>>
>> I could really use some incite here, this is a production website and
>> I need to act quickly. I have a cron job that takes a snapshot every
>> night, but even with that I tried to restore a snapshot on my local
>> development environment and it was also missing a ton of data.
>>
>> Any help will be so appreciated.
>>
>>
>>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com

-- 
Ben Coverston
DataStax -- The Apache Cassandra Company
http://www.datastax.com/


Mime
View raw message