cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Problem on node join the ring
Date Tue, 29 Jan 2013 07:55:20 GMT
>  there is no streaming anymore
Nodes only bootstrap once, when they are first started. 

> I have turned on the debug, this is what it is doing now(cpu is pretty much idle), no
any error message. 
Looks like it is receiving writes and reads, looks like it's part of the ring. 

Is this ring output from the Joining node or from one of the others ? Do the other nodes
see this node as up or joining ? 

When starting the node was there a log line with "Bootstrap variables" ? 

Anyways I would try running a nodetool repair -pr on the joining node. If you are not using
QUOURM / QUOURM you maybe getting inconsistent results now. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 29/01/2013, at 9:51 AM, Daning Wang <daning@netseer.com> wrote:

> I add a new node to ring(version 1.1.6), after more than 30 hours, it is still in the
'Joining' state
> 
> Address         DC          Rack        Status State   Load            Effective-Ownership
Token                                       
>                                                                                     
      141784319550391026443072753096570088105     
> 10.28.78.123    datacenter1 rack1       Up     Normal  18.73 GB        50.00%       
      0                                           
> 10.4.17.138     datacenter1 rack1       Up     Normal  15 GB           39.29%       
      24305883351495604533098186245126300818      
> 10.93.95.51     datacenter1 rack1       Up     Normal  17.96 GB        41.67%       
      42535295865117307932921825928971026432      
> 10.170.1.26     datacenter1 rack1       Up     Joining 6.89 GB         0.00%        
      56713727820156410577229101238628035242      
> 10.6.115.239    datacenter1 rack1       Up     Normal  20.3 GB         50.00%       
      85070591730234615865843651857942052864      
> 10.28.20.200    datacenter1 rack1       Up     Normal  22.68 GB        60.71%       
      127605887595351923798765477786913079296     
> 10.240.113.171  datacenter1 rack1       Up     Normal  18.4 GB         58.33%       
      141784319550391026443072753096570088105  
> 
> 
> since after a while, the cpu usage goes down to 0, looks it is stuck. I have restarted
server several times in last 30 hours. when server is just started, you can see streaming
in 'nodetool netstats', but after a few minutes, there is no streaming anymore
> 
> I have turned on the debug, this is what it is doing now(cpu is pretty much idle), no
any error message. 
> 
> Please help, I can provide more info if needed.
> 
> Thanks in advance,
> 
> 
> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 RowMutationVerbHandler.java (line 44)
Applying RowMutation(keyspace='dsat', key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079725000!3888000,])])
> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 Table.java (line 395) applying mutation
of row 52f5298affbb8bf0
> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 RowMutationVerbHandler.java (line 56)
RowMutation(keyspace='dsat', key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079725000!3888000,])]) applied.  Sending response to 571645593@/10.28.78.123
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 RowMutationVerbHandler.java (line 44)
Applying RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079730002,path:false:30@1359406079730001,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])])
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 395) applying mutation
of row 57f700499922964b
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 429) mutating indexed
column top_node value 6d617474626f7574726f732e74756d626c722e636f6d
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 CollationController.java (line 78) collectTimeOrderedData
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 453) Pre-mutation index
row is null
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 KeysIndex.java (line 119) applying index
row mattboutros.tumblr.com in ColumnFamily(dsatcache.dsatcache_top_node_idx [57f700499922964b:false:0@1359406079730000,])
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java (line 56)
RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079730002,path:false:30@1359406079730001,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])])
applied.  Sending response to 710680715@/10.28.20.200
> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java (line 44)
Applying RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079731000!3888000,])])
> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 Table.java (line 395) applying mutation
of row 57f700499922964b
> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java (line 56)
RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079731000!3888000,])]) applied.  Sending response to 710680719@/10.28.20.200
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 RowMutationVerbHandler.java (line 44)
Applying RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079692002,path:false:26@1359406079692001,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])])
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 395) applying mutation
of row 2a50083d5332071f
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 429) mutating indexed
column top_node value 7777772e706163696669632d72652e636f6d
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 CollationController.java (line 78) collectTimeOrderedData
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 453) Pre-mutation index
row is null
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 KeysIndex.java (line 119) applying index
row www.pacific-re.com in ColumnFamily(dsatcache.dsatcache_top_node_idx [2a50083d5332071f:false:0@1359406079692000,])
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java (line 56)
RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079692002,path:false:26@1359406079692001,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])])
applied.  Sending response to 710680779@/10.28.20.200
> DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java (line 44)
Applying RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079693000!3888000,])])
> DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 Table.java (line 395) applying mutation
of row 2a50083d5332071f
> DEBUG [MutationStage:6] 2013-01-28 12:47:59,654 RowMutationVerbHandler.java (line 56)
RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079693000!3888000,])]) applied.  Sending response to 710680784@/10.28.20.200
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 RowMutationVerbHandler.java (line 44)
Applying RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079783002,path:false:30@1359406079783001,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])])
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 395) applying mutation
of row 66e348eeeb00bc33
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 429) mutating indexed
column top_node value 63686963616964696f74612e74756d626c722e636f6d
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 CollationController.java (line 78) collectTimeOrderedData
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 453) Pre-mutation index
row is null
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 KeysIndex.java (line 119) applying index
row chicaidiota.tumblr.com in ColumnFamily(dsatcache.dsatcache_top_node_idx [66e348eeeb00bc33:false:0@1359406079783000,])
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 RowMutationVerbHandler.java (line 56)
RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079783002,path:false:30@1359406079783001,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])])
applied.  Sending response to 571645657@/10.28.78.123
> DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 RowMutationVerbHandler.java (line 44)
Applying RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[_meta:false:230@1359406079785000!3888000,])])
> DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 Table.java (line 395) applying mutation
of row 66e348eeeb00bc33
> DEBUG [MutationStage:16] 2013-01-28 12:47:59,679 RowMutationVerbHandler.java (line 56)
RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[_meta:false:230@1359406079785000!3888000,])]) applied.  Sending response to 571645662@/10.28.78.123
> 
> 
> 
> 


Mime
View raw message