incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Problem on node join the ring
Date Wed, 30 Jan 2013 23:05:44 GMT
erg, that error means it's not really part of the ring. 

I would try to restart the joining. 

Shut down the node, and delete everything in /var/lib/data/system. You can leave the data
that's already there if you want or delete it. 

Then try joining again. 

Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 30/01/2013, at 5:40 AM, Daning Wang <daning@netseer.com> wrote:

> Thanks very much Aaron.
> 
> * Other nodes still report it is in "Joining"
> * Here are bootstrap information in the log
> 
> [cassy@dsat305e.prod:/usr/local/cassy log]$ grep -i boot system.log
>  INFO [main] 2013-01-28 20:16:07,488 StorageService.java (line 774) JOINING: schema complete,
ready to bootstrap
>  INFO [main] 2013-01-28 20:16:07,489 StorageService.java (line 774) JOINING: getting
bootstrap token
>  INFO [main] 2013-01-28 20:16:37,518 StorageService.java (line 774) JOINING: Starting
to bootstrap...
> 
> * I tried to run repair -pr, but it gives exception
> 
> [cassy@dsat305e.prod:/usr/local/cassy log]$ nodetool -h localhost repair -pr
> Exception in thread "main" java.lang.AssertionError
>         at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:304)
>         at org.apache.cassandra.service.StorageService.getPrimaryRangeForEndpoint(StorageService.java:2080)
>         at org.apache.cassandra.service.StorageService.getLocalPrimaryRange(StorageService.java:211)
>         at org.apache.cassandra.service.StorageService.forceTableRepairPrimaryRange(StorageService.java:1993)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
>         at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
>         at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:235)
>         at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
>         at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:250)
>         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
>  
> 
> 
> On Mon, Jan 28, 2013 at 11:55 PM, aaron morton <aaron@thelastpickle.com> wrote:
>>  there is no streaming anymore
> Nodes only bootstrap once, when they are first started. 
> 
>> I have turned on the debug, this is what it is doing now(cpu is pretty much idle),
no any error message. 
> 
> Looks like it is receiving writes and reads, looks like it's part of the ring. 
> 
> Is this ring output from the Joining node or from one of the others ? Do the other nodes
> see this node as up or joining ? 
> 
> When starting the node was there a log line with "Bootstrap variables" ? 
> 
> Anyways I would try running a nodetool repair -pr on the joining node. If you are not
using QUOURM / QUOURM you maybe getting inconsistent results now. 
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 29/01/2013, at 9:51 AM, Daning Wang <daning@netseer.com> wrote:
> 
>> I add a new node to ring(version 1.1.6), after more than 30 hours, it is still in
the 'Joining' state
>> 
>> Address         DC          Rack        Status State   Load            Effective-Ownership
Token                                       
>>                                                                                 
          141784319550391026443072753096570088105     
>> 10.28.78.123    datacenter1 rack1       Up     Normal  18.73 GB        50.00%   
          0                                           
>> 10.4.17.138     datacenter1 rack1       Up     Normal  15 GB           39.29%   
          24305883351495604533098186245126300818      
>> 10.93.95.51     datacenter1 rack1       Up     Normal  17.96 GB        41.67%   
          42535295865117307932921825928971026432      
>> 10.170.1.26     datacenter1 rack1       Up     Joining 6.89 GB         0.00%    
          56713727820156410577229101238628035242      
>> 10.6.115.239    datacenter1 rack1       Up     Normal  20.3 GB         50.00%   
          85070591730234615865843651857942052864      
>> 10.28.20.200    datacenter1 rack1       Up     Normal  22.68 GB        60.71%   
          127605887595351923798765477786913079296     
>> 10.240.113.171  datacenter1 rack1       Up     Normal  18.4 GB         58.33%   
          141784319550391026443072753096570088105  
>> 
>> 
>> since after a while, the cpu usage goes down to 0, looks it is stuck. I have restarted
server several times in last 30 hours. when server is just started, you can see streaming
in 'nodetool netstats', but after a few minutes, there is no streaming anymore
>> 
>> I have turned on the debug, this is what it is doing now(cpu is pretty much idle),
no any error message. 
>> 
>> Please help, I can provide more info if needed.
>> 
>> Thanks in advance,
>> 
>> 
>> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 RowMutationVerbHandler.java (line
44) Applying RowMutation(keyspace='dsat', key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079725000!3888000,])])
>> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 Table.java (line 395) applying mutation
of row 52f5298affbb8bf0
>> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 RowMutationVerbHandler.java (line
56) RowMutation(keyspace='dsat', key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079725000!3888000,])]) applied.  Sending response to 571645593@/10.28.78.123
>> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 RowMutationVerbHandler.java (line
44) Applying RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079730002,path:false:30@1359406079730001,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])])
>> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 395) applying mutation
of row 57f700499922964b
>> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 429) mutating indexed
column top_node value 6d617474626f7574726f732e74756d626c722e636f6d
>> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 CollationController.java (line 78)
collectTimeOrderedData
>> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 453) Pre-mutation
index row is null
>> DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 KeysIndex.java (line 119) applying
index row mattboutros.tumblr.com in ColumnFamily(dsatcache.dsatcache_top_node_idx [57f700499922964b:false:0@1359406079730000,])
>> DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java (line
56) RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079730002,path:false:30@1359406079730001,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])])
applied.  Sending response to 710680715@/10.28.20.200
>> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java (line
44) Applying RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079731000!3888000,])])
>> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 Table.java (line 395) applying mutation
of row 57f700499922964b
>> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java (line
56) RowMutation(keyspace='dsat', key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079731000!3888000,])]) applied.  Sending response to 710680719@/10.28.20.200
>> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 RowMutationVerbHandler.java (line
44) Applying RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079692002,path:false:26@1359406079692001,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])])
>> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 395) applying mutation
of row 2a50083d5332071f
>> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 429) mutating indexed
column top_node value 7777772e706163696669632d72652e636f6d
>> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 CollationController.java (line 78)
collectTimeOrderedData
>> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 453) Pre-mutation
index row is null
>> DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 KeysIndex.java (line 119) applying
index row www.pacific-re.com in ColumnFamily(dsatcache.dsatcache_top_node_idx [2a50083d5332071f:false:0@1359406079692000,])
>> DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java (line
56) RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079692002,path:false:26@1359406079692001,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])])
applied.  Sending response to 710680779@/10.28.20.200
>> DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java (line
44) Applying RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079693000!3888000,])])
>> DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 Table.java (line 395) applying mutation
of row 2a50083d5332071f
>> DEBUG [MutationStage:6] 2013-01-28 12:47:59,654 RowMutationVerbHandler.java (line
56) RowMutation(keyspace='dsat', key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079693000!3888000,])]) applied.  Sending response to 710680784@/10.28.20.200
>> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 RowMutationVerbHandler.java (line
44) Applying RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079783002,path:false:30@1359406079783001,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])])
>> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 395) applying mutation
of row 66e348eeeb00bc33
>> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 429) mutating indexed
column top_node value 63686963616964696f74612e74756d626c722e636f6d
>> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 CollationController.java (line 78)
collectTimeOrderedData
>> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 453) Pre-mutation
index row is null
>> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 KeysIndex.java (line 119) applying
index row chicaidiota.tumblr.com in ColumnFamily(dsatcache.dsatcache_top_node_idx [66e348eeeb00bc33:false:0@1359406079783000,])
>> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 RowMutationVerbHandler.java (line
56) RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079783002,path:false:30@1359406079783001,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])])
applied.  Sending response to 571645657@/10.28.78.123
>> DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 RowMutationVerbHandler.java (line
44) Applying RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[_meta:false:230@1359406079785000!3888000,])])
>> DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 Table.java (line 395) applying mutation
of row 66e348eeeb00bc33
>> DEBUG [MutationStage:16] 2013-01-28 12:47:59,679 RowMutationVerbHandler.java (line
56) RowMutation(keyspace='dsat', key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[_meta:false:230@1359406079785000!3888000,])]) applied.  Sending response to 571645662@/10.28.78.123
>> 
>> 
>> 
>> 
> 
> 


Mime
View raw message