incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daning Wang <dan...@netseer.com>
Subject Re: Problem on node join the ring
Date Tue, 29 Jan 2013 16:40:41 GMT
Thanks very much Aaron.

* Other nodes still report it is in "Joining"
* Here are bootstrap information in the log

[cassy@dsat305e.prod:/usr/local/cassy log]$ grep -i boot system.log
 INFO [main] 2013-01-28 20:16:07,488 StorageService.java (line 774)
JOINING: schema complete, ready to bootstrap
 INFO [main] 2013-01-28 20:16:07,489 StorageService.java (line 774)
JOINING: getting bootstrap token
 INFO [main] 2013-01-28 20:16:37,518 StorageService.java (line 774)
JOINING: Starting to bootstrap...

* I tried to run repair -pr, but it gives exception

[cassy@dsat305e.prod:/usr/local/cassy log]$ nodetool -h localhost repair -pr
Exception in thread "main" java.lang.AssertionError
        at
org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:304)
        at
org.apache.cassandra.service.StorageService.getPrimaryRangeForEndpoint(StorageService.java:2080)
        at
org.apache.cassandra.service.StorageService.getLocalPrimaryRange(StorageService.java:211)
        at
org.apache.cassandra.service.StorageService.forceTableRepairPrimaryRange(StorageService.java:1993)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
        at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:235)
        at
com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:250)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)



On Mon, Jan 28, 2013 at 11:55 PM, aaron morton <aaron@thelastpickle.com>wrote:

>  there is no streaming anymore
>
> Nodes only bootstrap once, when they are first started.
>
> I have turned on the debug, this is what it is doing now(cpu is pretty
> much idle), no any error message.
>
> Looks like it is receiving writes and reads, looks like it's part of the
> ring.
>
> Is this ring output from the Joining node or from one of the others ? Do
> the other nodes
> see this node as up or joining ?
>
> When starting the node was there a log line with "Bootstrap variables" ?
>
> Anyways I would try running a nodetool repair -pr on the joining node. If
> you are not using QUOURM / QUOURM you maybe getting inconsistent results
> now.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 29/01/2013, at 9:51 AM, Daning Wang <daning@netseer.com> wrote:
>
> I add a new node to ring(version 1.1.6), after more than 30 hours, it is
> still in the 'Joining' state
>
> Address         DC          Rack        Status State   Load
>  Effective-Ownership Token
>
>                  141784319550391026443072753096570088105
> 10.28.78.123    datacenter1 rack1       Up     Normal  18.73 GB
>  50.00%              0
> 10.4.17.138     datacenter1 rack1       Up     Normal  15 GB
> 39.29%              24305883351495604533098186245126300818
> 10.93.95.51     datacenter1 rack1       Up     Normal  17.96 GB
>  41.67%              42535295865117307932921825928971026432
> 10.170.1.26     datacenter1 rack1       Up     Joining 6.89 GB
> 0.00%               56713727820156410577229101238628035242
> 10.6.115.239    datacenter1 rack1       Up     Normal  20.3 GB
> 50.00%              85070591730234615865843651857942052864
> 10.28.20.200    datacenter1 rack1       Up     Normal  22.68 GB
>  60.71%              127605887595351923798765477786913079296
> 10.240.113.171  datacenter1 rack1       Up     Normal  18.4 GB
> 58.33%              141784319550391026443072753096570088105
>
>
> since after a while, the cpu usage goes down to 0, looks it is stuck. I
> have restarted server several times in last 30 hours. when server is just
> started, you can see streaming in 'nodetool netstats', but after a few
> minutes, there is no streaming anymore
>
> I have turned on the debug, this is what it is doing now(cpu is pretty
> much idle), no any error message.
>
> Please help, I can provide more info if needed.
>
> Thanks in advance,
>
>
> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079725000!3888000,])])
> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 Table.java (line 395)
> applying mutation of row 52f5298affbb8bf0
> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079725000!3888000,])]) applied.  Sending response
> to 571645593@/10.28.78.123
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='57f700499922964b', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079730002,path:false:30@1359406079730001
> ,top_node:false:22@1359406079730000,v0:false:976@1359406079730003
> !3888000,])])
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 395)
> applying mutation of row 57f700499922964b
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 429)
> mutating indexed column top_node value
> 6d617474626f7574726f732e74756d626c722e636f6d
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 CollationController.java
> (line 78) collectTimeOrderedData
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 453)
> Pre-mutation index row is null
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 KeysIndex.java (line 119)
> applying index row mattboutros.tumblr.com in
> ColumnFamily(dsatcache.dsatcache_top_node_idx
> [57f700499922964b:false:0@1359406079730000,])
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,624
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='57f700499922964b', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079730002,path:false:30@1359406079730001
> ,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])])
> applied.  Sending response to 710680715@/10.28.20.200
> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='57f700499922964b', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079731000!3888000,])])
> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 Table.java (line 395)
> applying mutation of row 57f700499922964b
> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='57f700499922964b', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079731000!3888000,])]) applied.  Sending response
> to 710680719@/10.28.20.200
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079692002,path:false:26@1359406079692001
> ,top_node:false:18@1359406079692000,v0:false:583@1359406079692003
> !3888000,])])
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 395)
> applying mutation of row 2a50083d5332071f
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 429)
> mutating indexed column top_node value 7777772e706163696669632d72652e636f6d
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 CollationController.java
> (line 78) collectTimeOrderedData
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 453)
> Pre-mutation index row is null
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 KeysIndex.java (line 119)
> applying index row www.pacific-re.com in
> ColumnFamily(dsatcache.dsatcache_top_node_idx
> [2a50083d5332071f:false:0@1359406079692000,])
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,653
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079692002,path:false:26@1359406079692001
> ,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])])
> applied.  Sending response to 710680779@/10.28.20.200
> DEBUG [MutationStage:6] 2013-01-28 12:47:59,653
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079693000!3888000,])])
> DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 Table.java (line 395)
> applying mutation of row 2a50083d5332071f
> DEBUG [MutationStage:6] 2013-01-28 12:47:59,654
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079693000!3888000,])]) applied.  Sending response
> to 710680784@/10.28.20.200
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079783002,path:false:30@1359406079783001
> ,top_node:false:22@1359406079783000,v0:false:246@1359406079783003
> !3888000,])])
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 395)
> applying mutation of row 66e348eeeb00bc33
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 429)
> mutating indexed column top_node value
> 63686963616964696f74612e74756d626c722e636f6d
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 CollationController.java
> (line 78) collectTimeOrderedData
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 453)
> Pre-mutation index row is null
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 KeysIndex.java (line 119)
> applying index row chicaidiota.tumblr.com in
> ColumnFamily(dsatcache.dsatcache_top_node_idx
> [66e348eeeb00bc33:false:0@1359406079783000,])
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079783002,path:false:30@1359406079783001
> ,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])])
> applied.  Sending response to 571645657@/10.28.78.123
> DEBUG [MutationStage:16] 2013-01-28 12:47:59,678
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
> [_meta:false:230@1359406079785000!3888000,])])
> DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 Table.java (line 395)
> applying mutation of row 66e348eeeb00bc33
> DEBUG [MutationStage:16] 2013-01-28 12:47:59,679
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
> [_meta:false:230@1359406079785000!3888000,])]) applied.  Sending response
> to 571645662@/10.28.78.123
>
>
>
>
>
>

Mime
View raw message