Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of patrik.modesto@gmail.com
 designates 209.85.216.179 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=vCO5NWANBCf2xYg2T5utuTijo2ScMP46g7MPYrCfmdVZE2OQREDm+cdCkTzpsogsAK
         XZwQ7/nDNTMpFSGbQjIWimHUzU4cLT9RgjYz0qFSHIWiLlnByt9C7EPCl+ZzC8FbwIfT
         MZNe5YH52LhVVjmxArhBuasm01za5kQo3dLvQ=
MIME-Version: 1.0
From: Patrik Modesto <patrik.modesto@gmail.com>
Date: Tue, 15 Mar 2011 14:20:55 +0100
Message-ID: <AANLkTi=htKvMSU10eRVHGJJzsxJwWRSDGC6_DMZz0ijp@mail.gmail.com>
Subject: A way to break the cluster
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I'm did break my test cluster again. It's really strange. I use
cassandra 0.7.3. This is what I did:

- install node1
- install node2, auto_bootstrap: true
- install node3, auto_bootstrap: true

- created a keyspace with RF 1, populate with data
- create a keyspace with RF 3, populate with data

- nodetool -h node1 loadbalance - after an hour of doing nothing I
killed the nodetool, nodetool ring shows node1 as Up/Leaving
- restarted node1
- nodetool ring shows node1 as Up/Normal

>From now on nodetool loadbalance or decommission failt with:

nodetool -h skhd1.dev move 0
Exception in thread "main" java.lang.IllegalStateException:
replication factor (3) exceeds number of endpoints (2)
        at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60)
        at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929)
        at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895)
        at org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1595)
        at org.apache.cassandra.service.StorageService.move(StorageService.java:1733)
        at org.apache.cassandra.service.StorageService.move(StorageService.java:1708)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
        at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
        at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
        at sun.rmi.transport.Transport$1.run(Transport.java:159)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

nodetool repair pass, but nothing changed.

Regards,
Patrik