incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yan Chunlu <springri...@gmail.com>
Subject move one node for load re-balancing then it status stuck at "Leaving"
Date Fri, 05 Aug 2011 03:45:01 GMT
I have 3 nodes and the RF used to be 2, after awhile I have changed it
to 3;  using Cassandra 0.7.4
I have tried the nodetool move but get the following error....
node3:~# nodetool -h node3 move 0
Exception in thread "main" java.lang.IllegalStateException:
replication factor (3) exceeds number of endpoints (2)
at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60)
at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:930)
at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:896)
at org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1596)
at org.apache.cassandra.service.StorageService.move(StorageService.java:1734)
at org.apache.cassandra.service.StorageService.move(StorageService.java:1709)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)



then nodetool shows the node is leaving....
nodetool -h node3 ring
Address         Status State   Load            Owns    Token

84944475733633104818662955375549269696
node1      Up     Normal  13.18 GB        81.09%
52773518586096316348543097376923124102
node2     Up     Normal  22.85 GB        10.48%
70597222385644499881390884416714081360
node3      Up     Leaving 25.44 GB        8.43%
84944475733633104818662955375549269696


after go through the code I found the following code:
    /**
     * iterator over the Tokens in the given ring, starting with the
token for the node owning start
     * (which does not have to be a Token in the ring)
     * @param includeMin True if the minimum token should be returned
in the ring even if it has no owner.
     */
    public static Iterator<Token> ringIterator(final ArrayList<Token>
ring, Token start, boolean includeMin)



does "starting with the token for the node owning start" means I need
to move node1 at first?   what should I do now?  restart node3 and
start over?

why does it stuck at "Leaving" anyway?   it supposed to do or not do
it, not just stuck on the way......

Mime
View raw message