cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yan Chunlu <>
Subject move one node for load re-balancing then it status stuck at "Leaving"
Date Fri, 05 Aug 2011 03:45:01 GMT
I have 3 nodes and the RF used to be 2, after awhile I have changed it
to 3;  using Cassandra 0.7.4
I have tried the nodetool move but get the following error....
node3:~# nodetool -h node3 move 0
Exception in thread "main" java.lang.IllegalStateException:
replication factor (3) exceeds number of endpoints (2)
at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(
at org.apache.cassandra.service.StorageService.calculatePendingRanges(
at org.apache.cassandra.service.StorageService.calculatePendingRanges(
at org.apache.cassandra.service.StorageService.startLeaving(
at org.apache.cassandra.service.StorageService.move(
at org.apache.cassandra.service.StorageService.move(
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
at java.lang.reflect.Method.invoke(
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(
at com.sun.jmx.mbeanserver.PerInterface.invoke(
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
at java.lang.reflect.Method.invoke(
at sun.rmi.server.UnicastServerRef.dispatch(
at sun.rmi.transport.Transport$
at Method)
at sun.rmi.transport.Transport.serviceCall(
at sun.rmi.transport.tcp.TCPTransport.handleMessages(
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(
at sun.rmi.transport.tcp.TCPTransport$
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
at java.util.concurrent.ThreadPoolExecutor$

then nodetool shows the node is leaving....
nodetool -h node3 ring
Address         Status State   Load            Owns    Token

node1      Up     Normal  13.18 GB        81.09%
node2     Up     Normal  22.85 GB        10.48%
node3      Up     Leaving 25.44 GB        8.43%

after go through the code I found the following code:
     * iterator over the Tokens in the given ring, starting with the
token for the node owning start
     * (which does not have to be a Token in the ring)
     * @param includeMin True if the minimum token should be returned
in the ring even if it has no owner.
    public static Iterator<Token> ringIterator(final ArrayList<Token>
ring, Token start, boolean includeMin)

does "starting with the token for the node owning start" means I need
to move node1 at first?   what should I do now?  restart node3 and
start over?

why does it stuck at "Leaving" anyway?   it supposed to do or not do
it, not just stuck on the way......

View raw message