Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 8 Apr 2011 13:53:05 +0000 (UTC)
From: "Sasha Dolgy (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: 
 <355995797.43807.1302270785952.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <629685621.32253.1301919305725.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (CASSANDRA-2415) Ec2Snitch changing tokens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017457#comment-13017457 ] 

Sasha Dolgy commented on CASSANDRA-2415:
----------------------------------------

not seen anything since ... so maybe this is attributed to user error and not waiting long enough between bringing new nodes into the ring .. 

> Ec2Snitch changing tokens
> -------------------------
>
>                 Key: CASSANDRA-2415
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.4
>         Environment: Amazon EC2 -- 4 nodes
>            Reporter: Sasha Dolgy
>            Assignee: Brandon Williams
>            Priority: Minor
>             Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1.  Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2.  Brought up a second node, with the first node defined as the seed.  No visible issues.  
> 3.  Brought up node 3 in the same manner.  Receiving errors as shown in the output below.  
> Initially, I -did not- define tokens for the nodes. 
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$  INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
>  INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
>  INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798 
>         at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
>         at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
>         at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
>         at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
>         at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
>         at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
>         at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
>         at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
>         at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
>         at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
>         at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
>         at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
>         at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
>         at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> :~$  INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798.  /10.0.0.2 is the new owner
>  WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address         Status State   Load            Owns    Token
> 170141183460469231731687303715884105726
> 10.0.0.1    Up     Normal  99.31 KB        0.00%   0
> 10.0.0.2   Up     Normal  122.67 KB       11.22% 19095547144942516281182777765338228798
> 10.0.0.4   Up     Normal  103.75 KB       88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira