cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jens Rantil (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-8318) Unable to replace a node
Date Fri, 14 Nov 2014 20:04:36 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14212751#comment-14212751
] 

Jens Rantil edited comment on CASSANDRA-8318 at 11/14/14 8:04 PM:
------------------------------------------------------------------

bq. What actual token does .51 have?

It has a token of -2.

{noformat}
$ nodetool ring
Note: Ownership information does not include topology; for complete information, specify a
keyspace

Datacenter: Analytics
==========
Address     Rack        Status State   Load            Owns                Token
                                                                           6148914691236510000
X.X.X.52  rack1       Up     Normal  18.34 GB        0.02%               -9223372036854770000
X.X.X.50  rack1       Up     Normal  18.36 GB        0.24%               -6148914691236510000
X.X.X.55  rack1       Up     Normal  18.51 GB        0.19%               -3074457345618250000
X.X.X.51  rack1       Down   Normal  195.67 KB       0.02%               -2
X.X.X.54  rack1       Up     Normal  19.09 GB        0.04%               3074457345618258600
X.X.X.53  rack1       Up     Normal  18.5 GB         0.07%               6148914691236510000

Datacenter: Cassandra
==========
Address     Rack        Status State   Load            Owns                Token
                                                                           9219239585832170071
...

  Warning: "nodetool ring" is used to output all the tokens of a node.
  To view status related info of a node use "nodetool status" instead.
{noformat}


was (Author: ztyx):
> What actual token does .51 have?

It has a token of -2.

{noformat}
$ nodetool ring
Note: Ownership information does not include topology; for complete information, specify a
keyspace

Datacenter: Analytics
==========
Address     Rack        Status State   Load            Owns                Token
                                                                           6148914691236510000
X.X.X.52  rack1       Up     Normal  18.34 GB        0.02%               -9223372036854770000
X.X.X.50  rack1       Up     Normal  18.36 GB        0.24%               -6148914691236510000
X.X.X.55  rack1       Up     Normal  18.51 GB        0.19%               -3074457345618250000
X.X.X.51  rack1       Down   Normal  195.67 KB       0.02%               -2
X.X.X.54  rack1       Up     Normal  19.09 GB        0.04%               3074457345618258600
X.X.X.53  rack1       Up     Normal  18.5 GB         0.07%               6148914691236510000

Datacenter: Cassandra
==========
Address     Rack        Status State   Load            Owns                Token
                                                                           9219239585832170071
...

  Warning: "nodetool ring" is used to output all the tokens of a node.
  To view status related info of a node use "nodetool status" instead.
{noformat}

> Unable to replace a node
> ------------------------
>
>                 Key: CASSANDRA-8318
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8318
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: 2.0.8.39 (Datastax DSE 4.5.3)
>            Reporter: Jens Rantil
>         Attachments: X.X.X.56.log
>
>
> Had a hardware failure of a node. I followed the Datastax documentation[1] on how to
replace the node X.X.X.51 using a brand new node with the same IP. Since it didn't come up
after waiting for ~5 minutes or so, I decided to replace X.X.X.51 with a brand new unused
IP X.X.X.56 instead. It now seems like my gossip is some weird state. When I start the replacement
node I see line like
> {noformat}
>  INFO [GossipStage:1] 2014-11-14 14:57:03,025 Gossiper.java (line 901) InetAddress /X.X.X.51
is now DOWN
>  INFO [GossipStage:1] 2014-11-14 14:57:03,042 Gossiper.java (line 901) InetAddress /X.X.X.56
is now DOWN
> {noformat}
> . The latter is somewhat surprising since that is the IP of the actual replacement node.
It doesn't surprise me it can't talk to itself if it hasn't started!
> Eventually the replacement node shuts down with
> {noformat}
> ERROR [main] 2014-11-14 14:58:06,031 CassandraDaemon.java (line 513) Exception encountered
during startup
> java.lang.UnsupportedOperationException: Cannot replace token -2 which does not exist!
> 	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:782)
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:614)
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:503)
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
> 	at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:374)
> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> 	at com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:615)
>  INFO [Thread-2] 2014-11-14 14:58:06,035 DseDaemon.java (line 461) DSE shutting down...
>  INFO [StorageServiceShutdownHook] 2014-11-14 14:58:06,037 Gossiper.java (line 1307)
Announcing shutdown
>  INFO [Thread-2] 2014-11-14 14:58:06,046 PluginManager.java (line 355) All plugins are
stopped.
>  INFO [Thread-2] 2014-11-14 14:58:06,047 CassandraDaemon.java (line 463) Cassandra shutting
down...
> ERROR [Thread-2] 2014-11-14 14:58:06,047 CassandraDaemon.java (line 199) Exception in
thread Thread[Thread-2,5,main]
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.CassandraDaemon.stop(CassandraDaemon.java:464)
> 	at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:464)
> 	at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:364){noformat}
> All nodes are showing
> {noformat}
> root@machine-2:~# nodetool status company
> Datacenter: Analytics
> =====================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address     Load       Tokens  Owns (effective)  Host ID                        
      Rack
> UN  X.X.X.50  18.35 GB   1       16.7%             25efdbcd-14d3-4e9c-803a-3db5795d4efa
 rack1
> DN  X.X.X.51  195.67 KB  1       16.7%             d97cf86f-bfaf-4488-b716-26d71635a8fc
 rack1
> UN  X.X.X.52  18.7 GB    1       16.7%             caa32f68-5a6b-4d87-80bd-baa66a9b61ce
 rack1
> UN  X.X.X.53  18.56 GB   1       16.7%             e219321e-a6d5-48c4-9bad-d2e25429b1d2
 rack1
> UN  X.X.X.54  19.69 GB   1       16.7%             3cd36895-ee47-41c1-a5f5-41cb0f8526a6
 rack1
> UN  X.X.X.55  18.88 GB   1       16.7%             7d3f73c4-724e-45a6-bcf9-d3072dfc157f
 rack1
> Datacenter: Cassandra
> =====================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address     Load       Tokens  Owns (effective)  Host ID                        
      Rack
> UN  X.X.X.33  128.95 GB  256     100.0%            871968c9-1d6b-4f06-ba90-8b3a8d92dcf0
 rack1
> UN  X.X.X.32  115.3 GB   256     100.0%            d7cacd89-8613-4de5-8a5e-a2c53c41ea45
 rack1
> UN  X.X.X.31  130.45 GB  256     100.0%            48cb0782-6c9a-4805-9330-38e192b6b680
 rack1
> {noformat}
> , but when X.X.X.56 is starting is shows
> {noformat}
> root@machine-1:/var/lib/cassandra# nodetool status
> Note: Ownership information does not include topology; for complete information, specify
a keyspace
> Datacenter: Analytics
> =====================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address     Load       Tokens  Owns   Host ID                               Rack
> UN  X.X.X.50  18.41 GB   1       0.2%   25efdbcd-14d3-4e9c-803a-3db5795d4efa  rack1
> UN  X.X.X.52  19.07 GB   1       0.0%   caa32f68-5a6b-4d87-80bd-baa66a9b61ce  rack1
> UN  X.X.X.53  18.65 GB   1       0.1%   e219321e-a6d5-48c4-9bad-d2e25429b1d2  rack1
> UN  X.X.X.54  19.69 GB   1       0.0%   3cd36895-ee47-41c1-a5f5-41cb0f8526a6  rack1
> UN  X.X.X.55  18.97 GB   1       0.2%   7d3f73c4-724e-45a6-bcf9-d3072dfc157f  rack1
> Datacenter: Cassandra
> =====================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address     Load       Tokens  Owns   Host ID                               Rack
> UN  X.X.X.33  129.72 GB  256     21.7%  871968c9-1d6b-4f06-ba90-8b3a8d92dcf0  rack1
> UN  X.X.X.32  116 GB     256     12.4%  d7cacd89-8613-4de5-8a5e-a2c53c41ea45  rack1
> UN  X.X.X.31  130.62 GB  256     65.3%  48cb0782-6c9a-4805-9330-38e192b6b680  rack1
> {noformat}
> The above cluster state does not seem to replicate to the rest of the cluster (hasn't
so far).
> Any input on how I can restore world order is appreciated.
> [1] http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message