incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Janne Jalkanen <Janne.Jalka...@ecyrd.com>
Subject Re: Failed decommission
Date Sun, 25 Aug 2013 16:49:27 GMT

Thanks; this worked for me too.

/Janne

On Aug 25, 2013, at 18:47 , Mike Heffner <mike@librato.com> wrote:

> Janne,
> 
> We ran into this too. Appears it's a bug in 1.2.8 that is fixed in the upcoming 1.2.9.
I added the steps I took to finally remove the node here: https://issues.apache.org/jira/browse/CASSANDRA-5857?focusedCommentId=13748998&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13748998
> 
> 
> Cheers,
> 
> Mike
> 
> 
> On Sun, Aug 25, 2013 at 4:06 AM, Janne Jalkanen <Janne.Jalkanen@ecyrd.com> wrote:
> This on cass 1.2.8
> 
> Ring state before decommission
> 
> --  Address         Load       Owns   Host ID                               Token   
                                Rack
> UN  10.0.0.1  38.82 GB   33.3%  21a98502-dc74-4ad0-9689-0880aa110409  1             
                          1a
> UN  10.0.0.2   33.5 GB    33.3%  cba6b27a-4982-4f04-854d-cc73155d5f69  56713727820156407428984779325531226110
  1b
> UN  10.0.0.3  37.41 GB   0.0%   6ba2c7d4-713e-4c14-8df8-f861fb211b0d  56713727820156407428984779325531226111
  1b
> UN  10.0.0.4  35.7 GB    33.3%  bf3d4792-f3e0-4062-afe3-be292bc85ed7  113427455640312814857969558651062452222
 1c
> 
> Trying to decommission the node
> 
> ubuntu@10.0.0.3:~$ nodetool decommission
> Exception in thread "main" java.lang.NumberFormatException: For input string: "56713727820156407428984779325531226111"
>         at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>         at java.lang.Long.parseLong(Long.java:444)
>         at java.lang.Long.parseLong(Long.java:483)
>         at org.apache.cassandra.service.StorageService.extractExpireTime(StorageService.java:1660)
>         at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1515)
>         at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1234)
>         at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:949)
>         at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1116)
>         at org.apache.cassandra.service.StorageService.leaveRing(StorageService.java:2817)
>         at org.apache.cassandra.service.StorageService.unbootstrap(StorageService.java:2861)
>         at org.apache.cassandra.service.StorageService.decommission(StorageService.java:2808)
> 
> Now I'm in a state where the machine is still "up" but "leaving" but I can't seem to
get it out of the ring.  For example:
> 
> % nodetool removenode 6ba2c7d4-713e-4c14-8df8-f861fb211b0d
> Exception in thread "main" java.lang.UnsupportedOperationException: Node /10.0.0.3 is
alive and owns this ID. Use decommission command to remove it from the ring
> 
> Any ideas?
> 
> /Janne
> 
> 
> 
> -- 
> 
>   Mike Heffner <mike@librato.com>
>   Librato, Inc.
> 


Mime
View raw message