erg, that error means it's not really part of the ring. 

I would try to restart the joining. 

Shut down the node, and delete everything in /var/lib/data/system. You can leave the data that's already there if you want or delete it. 

Then try joining again. 

Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton

On 30/01/2013, at 5:40 AM, Daning Wang <daning@netseer.com> wrote:

Thanks very much Aaron.

* Other nodes still report it is in "Joining"
* Here are bootstrap information in the log

[cassy@dsat305e.prod:/usr/local/cassy log]$ grep -i boot system.log
 INFO [main] 2013-01-28 20:16:07,488 StorageService.java (line 774) JOINING: schema complete, ready to bootstrap
 INFO [main] 2013-01-28 20:16:07,489 StorageService.java (line 774) JOINING: getting bootstrap token
 INFO [main] 2013-01-28 20:16:37,518 StorageService.java (line 774) JOINING: Starting to bootstrap...

* I tried to run repair -pr, but it gives exception

[cassy@dsat305e.prod:/usr/local/cassy log]$ nodetool -h localhost repair -pr
Exception in thread "main" java.lang.AssertionError
        at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:304)
        at org.apache.cassandra.service.StorageService.getPrimaryRangeForEndpoint(StorageService.java:2080)
        at org.apache.cassandra.service.StorageService.getLocalPrimaryRange(StorageService.java:211)
        at org.apache.cassandra.service.StorageService.forceTableRepairPrimaryRange(StorageService.java:1993)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:235)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:250)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
 


On Mon, Jan 28, 2013 at 11:55 PM, aaron morton <aaron@thelastpickle.com> wrote:
 there is no streaming anymore
Nodes only bootstrap once, when they are first started. 

I have turned on the debug, this is what it is doing now(cpu is pretty much idle), no any error message. 
Looks like it is receiving writes and reads, looks like it's part of the ring. 

Is this ring output from the Joining node or from one of the others ? Do the other nodes
see this node as up or joining ? 

When starting the node was there a log line with "Bootstrap variables" ? 

Anyways I would try running a nodetool repair -pr on the joining node. If you are not using QUOURM / QUOURM you maybe getting inconsistent results now. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton

On 29/01/2013, at 9:51 AM, Daning Wang <daning@netseer.com> wrote:

I add a new node to ring(version 1.1.6), after more than 30 hours, it is still in the 'Joining' state

Address         DC          Rack        Status State   Load            Effective-Ownership Token                                       
                                                                                           141784319550391026443072753096570088105     
10.28.78.123    datacenter1 rack1       Up     Normal  18.73 GB        50.00%              0                                           
10.4.17.138     datacenter1 rack1       Up     Normal  15 GB           39.29%              24305883351495604533098186245126300818      
10.93.95.51     datacenter1 rack1       Up     Normal  17.96 GB        41.67%              42535295865117307932921825928971026432      
10.170.1.26     datacenter1 rack1       Up     Joining 6.89 GB         0.00%               56713727820156410577229101238628035242      
10.6.115.239    datacenter1 rack1       Up     Normal  20.3 GB         50.00%              85070591730234615865843651857942052864      
10.28.20.200    datacenter1 rack1       Up     Normal  22.68 GB        60.71%              127605887595351923798765477786913079296     
10.240.113.171  datacenter1 rack1       Up     Normal  18.4 GB         58.33%              141784319550391026443072753096570088105  


since after a while, the cpu usage goes down to 0, looks it is stuck. I have restarted server several times in last 30 hours. when server is just started, you can see streaming in 'nodetool netstats', but after a few minutes, there is no streaming anymore

I have turned on the debug, this is what it is doing now(cpu is pretty much idle), no any error message. 

Please help, I can provide more info if needed.

Thanks in advance,


DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat', key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache [_meta:false:278@1359406079725000!3888000,])])
DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 Table.java (line 395) applying mutation of row 52f5298affbb8bf0
DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat', key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache [_meta:false:278@1359406079725000!3888000,])]) applied.  Sending response to 571645593@/10.28.78.123
DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache [cache_type:false:8@1359406079730002,path:false:30@1359406079730001,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])])
DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 395) applying mutation of row 57f700499922964b
DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 429) mutating indexed column top_node value 6d617474626f7574726f732e74756d626c722e636f6d
DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 CollationController.java (line 78) collectTimeOrderedData
DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 453) Pre-mutation index row is null
DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 KeysIndex.java (line 119) applying index row mattboutros.tumblr.com in ColumnFamily(dsatcache.dsatcache_top_node_idx [57f700499922964b:false:0@1359406079730000,])
DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache [cache_type:false:8@1359406079730002,path:false:30@1359406079730001,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])]) applied.  Sending response to 710680715@/10.28.20.200
DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache [_meta:false:278@1359406079731000!3888000,])])
DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 Table.java (line 395) applying mutation of row 57f700499922964b
DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache [_meta:false:278@1359406079731000!3888000,])]) applied.  Sending response to 710680719@/10.28.20.200
DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache [cache_type:false:8@1359406079692002,path:false:26@1359406079692001,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])])
DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 395) applying mutation of row 2a50083d5332071f
DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 429) mutating indexed column top_node value 7777772e706163696669632d72652e636f6d
DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 CollationController.java (line 78) collectTimeOrderedData
DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 453) Pre-mutation index row is null
DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 KeysIndex.java (line 119) applying index row www.pacific-re.com in ColumnFamily(dsatcache.dsatcache_top_node_idx [2a50083d5332071f:false:0@1359406079692000,])
DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache [cache_type:false:8@1359406079692002,path:false:26@1359406079692001,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])]) applied.  Sending response to 710680779@/10.28.20.200
DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache [_meta:false:278@1359406079693000!3888000,])])
DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 Table.java (line 395) applying mutation of row 2a50083d5332071f
DEBUG [MutationStage:6] 2013-01-28 12:47:59,654 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache [_meta:false:278@1359406079693000!3888000,])]) applied.  Sending response to 710680784@/10.28.20.200
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache [cache_type:false:8@1359406079783002,path:false:30@1359406079783001,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])])
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 395) applying mutation of row 66e348eeeb00bc33
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 429) mutating indexed column top_node value 63686963616964696f74612e74756d626c722e636f6d
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 CollationController.java (line 78) collectTimeOrderedData
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 453) Pre-mutation index row is null
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 KeysIndex.java (line 119) applying index row chicaidiota.tumblr.com in ColumnFamily(dsatcache.dsatcache_top_node_idx [66e348eeeb00bc33:false:0@1359406079783000,])
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache [cache_type:false:8@1359406079783002,path:false:30@1359406079783001,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])]) applied.  Sending response to 571645657@/10.28.78.123
DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache [_meta:false:230@1359406079785000!3888000,])])
DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 Table.java (line 395) applying mutation of row 66e348eeeb00bc33
DEBUG [MutationStage:16] 2013-01-28 12:47:59,679 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache [_meta:false:230@1359406079785000!3888000,])]) applied.  Sending response to 571645662@/10.28.78.123