Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of sdolgy@gmail.com designates
 209.85.216.179 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=bFmTj6j8ZvPqGTJqFl9D6+jhdZ1/SRKsi5DCXalrIcN1g8g4Yibx2ukih9bgsBtKr+
         rQOf15ysFIcGv7Ul8ghrhfNPrpIR+U7Fj8yFQErXd2rjQ49xuts+ynr75BO65NeJrsqs
         H62Tdsss2O4KdK0UYg9aPL5pxFCH0SD9Zc1lI=
MIME-Version: 1.0
From: Sasha Dolgy <sdolgy@gmail.com>
Date: Wed, 23 Mar 2011 01:47:23 +0100
Message-ID: <AANLkTim9B5gtXppt7EadnvTY2H+fzhDUewNmhGkxZBA+@mail.gmail.com>
Subject: 0.7.4 problems .. snitch?
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi there,

Installed a new 4 node 0.7.4 cluster on ec2.  Brought up the first
node without issue with Ec2Snitch configured in the cassandra.yaml.

Brought up a second node, with the first node defined as the seed.  No
visible issues.  3 & 4 however are giving me problems as shown in the
output below.  Initially, I -did not- define tokens. When node 3 came
up, I had this error, so i went and manually moved the tokens and did
a nodetool move/repair/clean before getting on to node 4.

The tokens for the 4 nodes:

0
19095547144942516281182777765338228798
56713727820156410577229101238628035242
170141183460469231731687303715884105726

So now, when the 4th node comes online, with it's token set in the
cassandra.yaml (first one i did it for because of the errors I saw
with node 3) ... everything goes well at first, in joining the ring,
etc.....and then I see the following error in the system.log:

:~$  INFO [HintedHandoff:1] 2011-03-23 00:37:24,298
HintedHandOffManager.java (line 304) Started hinted handoff for
endpoint /10.0.0.2
 INFO [HintedHandoff:1] 2011-03-23 00:37:24,298
HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
to endpoint /10.0.0.2
 INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java
(line 702) Node /10.0.0.2 state jump to bootstrap
ERROR [GossipStage:2] 2011-03-23 00:37:55,381
DebuggableThreadPoolExecutor.java (line 103) Error in
ThreadPoolExecutor
java.lang.RuntimeException: Bootstrap Token collision between
/10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
        at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
        at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
        at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
        at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
        at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
        at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
        at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
ERROR [GossipStage:2] 2011-03-23 00:37:55,382
AbstractCassandraDaemon.java (line 112) Fatal exception in thread
Thread[GossipStage:2,5,main]
java.lang.RuntimeException: Bootstrap Token collision between
/10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
        at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
        at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
        at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
        at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
        at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
        at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
        at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

:~$  INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
(line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token
19095547144942516281182777765338228798.  /10.0.0.2 is the new owner
 WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
115) Token 19095547144942516281182777765338228798 changing ownership
from /10.0.0.3 to /10.0.0.2

:~$ nodetool -h 10.0.0.1 -p 9090 ring
Address         Status State   Load            Owns    Token

170141183460469231731687303715884105726
10.0.0.1    Up     Normal  99.31 KB        0.00%   0
10.0.0.2   Up     Normal  122.67 KB       11.22%
19095547144942516281182777765338228798
10.0.0.4   Up     Normal  103.75 KB       88.78%
170141183460469231731687303715884105726
:~$


Should I be a bit more hands off with the Ec2Snitch .... ?  Now i have
3 nodes with 1 having a duplicate token ....

-- 
Sasha Dolgy
sasha.dolgy@gmail.com