incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: new node gets no data
Date Fri, 16 Mar 2012 09:03:42 GMT
ahh, I think you may have hit a corner case here. 

If the RF still 1 ? 

> INFO [AntiEntropySessions:1] 2012-03-16 06:15:13,727
> AntiEntropyService.java (line 663) [repair #%s] No neighbors to repair
> with on range %s: session completed
Means there are no nodes which share the range with this node. So there is nothing to repair.


To put it another way: As far is 161.101 is concerned none of the keys it is responsible for
are stored on another node. So there are no other nodes that could be involved in a repair
session. 

It looks like some data may have been written to 161.101 so I think the safest approach would
be:
* increase the RF to 2
* repair
* decrease the RF to 1

When you added the node was auto_bootstrap enabled ? I would have thought that would stream
data from the first node to the new one. 

Cheers

  
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 16/03/2012, at 7:22 PM, Thorsten von Eicken wrote:

> Thanks for the suggestion Aaron, unfortunately, that seems to do
> absolutely nothing:
> 
> # nodetool -h localhost repair
>  INFO [RMI TCP Connection(160)-127.0.0.1] 2012-03-16 06:15:13,718
> StorageService.java (line 1770) Starting repair command #1, repairing 1
> ranges.
> INFO [AntiEntropySessions:1] 2012-03-16 06:15:13,727
> AntiEntropyService.java (line 658) [repair
> #6472b290-6f2f-11e1-0000-472739b10cff] new session: will sync
> /10.80.161.101 on range (0,85070591730234615865843651857942052864] for
> rslog_production.[users, req_text, req_attr_idx, req_word_idx,
> req_word_freq, sessions, requests, info]
> INFO [AntiEntropySessions:1] 2012-03-16 06:15:13,727
> AntiEntropyService.java (line 663) [repair #%s] No neighbors to repair
> with on range %s: session completed
> INFO [RMI TCP Connection(160)-127.0.0.1] 2012-03-16 06:15:13,727
> StorageService.java (line 1807) Repair command #1 completed successfully
> 
> Stumped...
>    TvE
> 
> 
> On 3/15/2012 6:41 PM, aaron morton wrote:
>> trying running nodetool repair on 10.80.161.101 and then cleanup
>> on 10.102.37.168 if everything is ok. 
>> 
>> Cheers
>> 
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 16/03/2012, at 6:45 AM, Thorsten von Eicken wrote:
>> 
>>> I added a second node to a single-node ring. RF=1. I can't get the new
>>> node to receive any data. Logs look fine. Here's what nodetool reports:
>>> 
>>> # nodetool -h localhost ring
>>> Address         DC          Rack        Status State   Load           
>>> Owns    Token
>>> 
>>> 85070591730234615865843651857942052864
>>> 10.102.37.168   datacenter1 rack1       Up     Normal  807.81 GB      
>>> 50.00%  0
>>> 10.80.161.101   datacenter1 rack1       Up     Normal  1.15 MB        
>>> 50.00%  85070591730234615865843651857942052864
>>> 
>>> Just a "little" imbalance. Yes, I use partitioner:
>>> org.apache.cassandra.dht.RandomPartitioner
>>> I tried moving the new node's token up/down by 1 and it triggers the log
>>> messages you'd expect, but no data gets transferred. How do I
>>> troubleshoot this? Below are the log messages I see when restarting the
>>> new node:
>>> 
>>> INFO [main] 2012-03-15 17:31:08,616 AbstractCassandraDaemon.java (line
>>> 120) JVM vendor/version:
>>> Java HotSpot(TM) 64-Bit Server VM/1.6.0_24
>>> INFO [main] 2012-03-15 17:31:14,812 CommitLog.java (line 178) Log
>>> replay complete, 8 replayed mutations
>>> INFO [main] 2012-03-15 17:31:14,825 StorageService.java (line 390)
>>> Cassandra version: 1.0.6
>>> INFO [main] 2012-03-15 17:31:14,825 StorageService.java (line 391)
>>> Thrift API version: 19.19.0
>>> INFO [main] 2012-03-15 17:31:14,825 StorageService.java (line 404)
>>> Loading persisted ring state
>>> INFO [main] 2012-03-15 17:31:14,834 StorageService.java (line 482)
>>> Starting up server gossip
>>> INFO [main] 2012-03-15 17:31:15,372 MessagingService.java (line 247)
>>> Starting Encrypted Messaging Service on SSL port 7000
>>> INFO [main] 2012-03-15 17:31:15,376 MessagingService.java (line 268)
>>> Starting Messaging Service on port 7001
>>> INFO [main] 2012-03-15 17:31:15,401 StorageService.java (line 579)
>>> Using saved token 85070591730234615865843651857942052864
>>> INFO [main] 2012-03-15 17:31:15,402 ColumnFamilyStore.java (line 692)
>>> Enqueuing flush of Memtable-LocationInfo@645492252(53/66 serialized/live
>>> bytes, 2 ops)
>>> INFO [FlushWriter:1] 2012-03-15 17:31:15,403 Memtable.java (line 240)
>>> Writing Memtable-LocationInfo@645492252(53/66 serialized/live bytes,
>>> 2 ops)
>>> INFO [FlushWriter:1] 2012-03-15 17:31:15,421 Memtable.java (line 277)
>>> Completed flushing /mnt/ebs/data/system/LocationInfo-hc-32-Data.db (163
>>> bytes)
>>> INFO [main] 2012-03-15 17:31:15,424 StorageService.java (line 948) Node
>>> /10.80.161.101 state jump to normal
>>> INFO [main] 2012-03-15 17:31:15,434 StorageService.java (line 589)
>>> Bootstrap/Replace/Move completed! Now serving reads.
>>> 
>>> # describe keyspace
>>> Keyspace: rslog_production:
>>> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
>>> Durable Writes: true
>>>   Options: [replication_factor:1]
>>> Column Families:
>>> 
>> 


Mime
View raw message