cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: cluster issues
Date Tue, 29 Jan 2013 07:15:45 GMT
> We can always be proactive in keeping the time sync. But, Is there any way to recover
from a time drift (in a reactive manner)? Since it was a lab environment, I dropped the KS
(deleted data directory)
There is a way to remove future dated columns, but it not for the faint hearted. 

Basically:
1) Drop the gc_grace_seconds to 0
2) Delete the column with a timestamp way in the future, so it is guaranteed to be higher
than the value you want to delete. 
3) Flush the CF
4) Compact all the SSTables that contain the row. The easiest way to do that is a major compaction,
but we normally advise not to do that because it creates one big file. You can also do a user
defined compaction. 

> Are there any other scenarios that would lead a cluster look like below? Note:Actual
topology of the cluster - ONE Cassandra node and TWO Analytic nodes.
What snitch are you using?
If you have the property file snitch do all nodes have the same configuration ?

There is a lot of sickness there. If possible I would scrub and start again. 

Cheers
 
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 29/01/2013, at 6:29 AM, S C <asf11@outlook.com> wrote:

> One of our node in a 3 node cluster drifted by ~ 20-25 seconds. While I figured this
pretty quickly, I had few questions that am looking for some answers.
> 
> We can always be proactive in keeping the time sync. But, Is there any way to recover
from a time drift (in a reactive manner)? Since it was a lab environment, I dropped the KS
(deleted data directory).
> Are there any other scenarios that would lead a cluster look like below? Note:Actual
topology of the cluster - ONE Cassandra node and TWO Analytic nodes.
> 
> 
> On 192.168.2.100
> Address         DC          Rack        Status State   Load            Owns         
      Token                                       
>                                                                                     
      113427455640312821154458202477256070485     
> 192.168.2.100  Cassandra   rack1       Up     Normal  601.34 MB       33.33%        
     0                                           
> 192.168.2.101  Analytics   rack1       Down   Normal  149.75 MB       33.33%        
     56713727820156410577229101238628035242      
> 192.168.2.102  Analytics   rack1       Down   Normal  ?               33.33%        
     113427455640312821154458202477256070485   
> 
> On 192.168.2.101
> Address         DC          Rack        Status State   Load            Owns         
      Token                                       
>                                                                                     
      113427455640312821154458202477256070485     
> 192.168.2.100  Analytics   rack1       Down   Normal  ?               33.33%        
     0                                           
> 192.168.2.101  Analytics   rack1       Up     Normal  158.59 MB       33.33%        
     56713727820156410577229101238628035242      
> 192.168.2.102  Analytics   rack1       Down   Normal  ?               33.33%        
     113427455640312821154458202477256070485    
> 
> On 192.168.2.102
> Address         DC          Rack        Status State   Load            Owns         
      Token                                       
>                                                                                     
      113427455640312821154458202477256070485     
> 192.168.2.100  Analytics   rack1       Down   Normal  ?               33.33%        
     0                                           
> 192.168.2.101  Analytics   rack1       Down   Normal  ?               33.33%        
     56713727820156410577229101238628035242      
> 192.168.2.102  Analytics   rack1       Up     Normal  117.02 MB       33.33%        
     113427455640312821154458202477256070485     
> 
> 
> Appreciate your valuable inputs.
> 
> Thanks,
> SC


Mime
View raw message