cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Colby <jonathan.co...@gmail.com>
Subject Re: exception when adding a node replication factor (3) exceeds number of endpoints (1) - SOLVED
Date Sat, 28 May 2011 10:29:43 GMT
OK, is seems a "phantom" node (one that was removed from the cluster)
kept being passed around in gossip as a down endpoint and was messing
up the gossip algorithm.  I had the luxury of being able to stop the
entire cluster and bring the nodes up one by one.  That purged the bad
node from gossip.  Not sure if there was a more elegant way to do
that.

On Fri, May 27, 2011 at 9:28 AM,  <jonathan.colby@gmail.com> wrote:
> Anyone have any idea what this could mean?
> This is a cluster of 7 nodes, I'm trying to add the 8th node.
>
> INFO [FlushWriter:1] 2011-05-27 09:22:40,495 Memtable.java (line 164)
> Completed flushing /var/lib/cassandra/data/system/Migrations-f-1-Data.db
> (6358 bytes)
> INFO [FlushWriter:1] 2011-05-27 09:22:40,496 Memtable.java (line 157)
> Writing Memtable-Schema@60230368(2363 bytes, 3 operations)
> INFO [FlushWriter:1] 2011-05-27 09:22:40,562 Memtable.java (line 164)
> Completed flushing /var/lib/cassandra/data/system/Schema-f-1-Data.db (2513
> bytes)
> INFO [GossipStage:1] 2011-05-27 09:22:40,829 Gossiper.java (line 610) Node
> /10.46.108.104 is now part of the cluster
> ERROR [GossipStage:1] 2011-05-27 09:22:40,845
> DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor
> java.lang.IllegalStateException: replication factor (3) exceeds number of
> endpoints (1)
> at
> org.apache.cassandra.locator.OldNetworkTopologyStrategy.calculateNaturalEndpoints(OldNetworkTopologyStrategy.java:100)
> at
> org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:196)
> at
> org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:945)
> at
> org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:896)
> at
> org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:707)
> at
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1124)
> at
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:643)
> at org.apache.cassandra.gms.Gossiper.handleNewJoin(Gossiper.java:611)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:690)
> at
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:1] 2011-05-27 09:22:40,847 AbstractCassandraDaemon.java
> (line 112) Fatal exception in thread Thread[GossipStage:1,5,main]
> java.lang.IllegalStateException: replication factor (3) exceeds number of
> endpoints (1)
> at
> org.apache.cassandra.locator.OldNetworkTopologyStrategy.calculateNaturalEndpoints(OldNetworkTopologyStrategy.java:100)
> at
> org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:196)
> at
> org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:945)
> at
> org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:896)
> at
> org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:707)
> at
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1124)
> at
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:643)
> at org.apache.cassandra.gms.Gossiper.handleNewJoin(Gossiper.java:611)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:690)
> at
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)

Mime
View raw message