cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mir Tanvir Hossain <mir.tanvir.hoss...@gmail.com>
Subject Re: Problem Replacing a Dead Node
Date Thu, 21 Apr 2016 18:02:22 GMT
Hi Anubhav, thanks for getting back to me. here is the information that you
requested.

datastax agent is running on the node. However, in the agent log I see

ERROR [clojure-agent-send-off-pool-4] 2016-04-21 17:51:46,055 Can't connect
to Cassandra (All host(s) tried for query failed (tried: /10.0.7.4:9042
(com.datastax.driver.core.TransportException: [/10.0.7.4:9042] Cannot
connect))), retrying soon.
ERROR [clojure-agent-send-off-pool-5] 2016-04-21 17:51:46,056 Can't connect
to Cassandra (All host(s) tried for query failed (tried: /10.0.7.4:9042
(com.datastax.driver.core.TransportException: [/10.0.7.4:9042] Cannot
connect))), retrying soon.

I am guessing this is because the node is not accepting any reads yet.


here is the output of nodetool status from the replacement node

mhossain@cassandra-24:~$ nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns   Host ID
    Rack
UN  10.0.7.80   117.32 GB  256     14.2%
 508b6503-e342-41bf-9baf-1f16ed1ebfc8  1a
UN  10.0.7.4    68.46 GB   256     15.1%
 b9a99507-ef83-441a-a007-da91144fae8f  1a
UN  10.0.7.190  106.54 GB  256     15.0%
 32cb119e-e13d-45db-89d1-a4385c47cee2  1a
UN  10.0.7.100  80.99 GB   256     13.5%
 efe3f327-48e8-4105-b096-a7f5c85736f9  1a
UN  10.0.7.195  105.9 GB   256     14.1%
 96403b7e-57fd-4b84-9607-745ec2d826df  1a
UN  10.0.7.160  98.42 GB   256     13.9%
 3a788a95-63f9-44f2-af91-9f49de75db63  1a
UN  10.0.7.176  93.04 GB   256     14.3%
 d9124ced-847d-474e-a230-ea67ba46dfa8  1a


here is the output of nodetool status from another existing node on the
cluster:

mhossain@cassandra-13:~$ nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns   Host ID
    Rack
UN  10.0.7.80   117.32 GB  256     14.2%
 508b6503-e342-41bf-9baf-1f16ed1ebfc8  1a
UN  10.0.7.190  106.54 GB  256     15.0%
 32cb119e-e13d-45db-89d1-a4385c47cee2  1a
UN  10.0.7.100  80.99 GB   256     13.5%
 efe3f327-48e8-4105-b096-a7f5c85736f9  1a
UN  10.0.7.195  105.9 GB   256     14.1%
 96403b7e-57fd-4b84-9607-745ec2d826df  1a
DN  10.0.7.91   115.97 GB  256     15.1%
 b9a99507-ef83-441a-a007-da91144fae8f  1a
UN  10.0.7.160  98.42 GB   256     13.9%
 3a788a95-63f9-44f2-af91-9f49de75db63  1a
UN  10.0.7.176  93.04 GB   256     14.3%
 d9124ced-847d-474e-a230-ea67ba46dfa8  1a


10.0.7.91 is the node I am trying to replace.


Here is the output of tail -n 50 /var/log/cassandra/system.log

INFO [GossipStage:1] 2016-04-21 17:58:01,812 StorageService.java (line
1665) Nodes /10.0.7.91 and /10.0.7.4 have the same token
8885324152940404221.  Ignoring /10.0.7.91
DEBUG [GossipStage:1] 2016-04-21 17:58:01,812 StorageService.java (line
1671) Relocating ranges:
 INFO [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1665) Nodes /10.0.7.91 and /10.0.7.4 have the same token
8894597858951527418.  Ignoring /10.0.7.91
DEBUG [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1671) Relocating ranges:
 INFO [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1665) Nodes /10.0.7.91 and /10.0.7.4 have the same token
8895886558199074220.  Ignoring /10.0.7.91
DEBUG [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1671) Relocating ranges:
 INFO [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1665) Nodes /10.0.7.91 and /10.0.7.4 have the same token
8943679396315445898.  Ignoring /10.0.7.91
DEBUG [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1671) Relocating ranges:
 INFO [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1665) Nodes /10.0.7.91 and /10.0.7.4 have the same token
8971093763454238578.  Ignoring /10.0.7.91
DEBUG [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1671) Relocating ranges:
 INFO [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1665) Nodes /10.0.7.91 and /10.0.7.4 have the same token
9147567932890414079.  Ignoring /10.0.7.91
DEBUG [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1671) Relocating ranges:
 INFO [GossipStage:1] 2016-04-21 17:58:01,813 StorageService.java (line
1665) Nodes /10.0.7.91 and /10.0.7.4 have the same token
9201669617284985565.  Ignoring /10.0.7.91
DEBUG [GossipStage:1] 2016-04-21 17:58:01,814 StorageService.java (line
1671) Relocating ranges:
 INFO [GossipStage:1] 2016-04-21 17:58:01,814 StorageService.java (line
1665) Nodes /10.0.7.91 and /10.0.7.4 have the same token
930880968512921941.  Ignoring /10.0.7.91
DEBUG [GossipStage:1] 2016-04-21 17:58:01,814 StorageService.java (line
1671) Relocating ranges:
DEBUG [PendingRangeCalculator:1] 2016-04-21 17:58:01,814
PendingRangeCalculatorService.java (line 128) No bootstrapping, leaving or
moving nodes, and no relocating tokens -> empty pending ranges for
test_shadow
DEBUG [PendingRangeCalculator:1] 2016-04-21 17:58:01,814
PendingRangeCalculatorService.java (line 128) No bootstrapping, leaving or
moving nodes, and no relocating tokens -> empty pending ranges for app
DEBUG [PendingRangeCalculator:1] 2016-04-21 17:58:01,814
PendingRangeCalculatorService.java (line 128) No bootstrapping, leaving or
moving nodes, and no relocating tokens -> empty pending ranges for test
DEBUG [PendingRangeCalculator:1] 2016-04-21 17:58:01,814
PendingRangeCalculatorService.java (line 128) No bootstrapping, leaving or
moving nodes, and no relocating tokens -> empty pending ranges for OpsCenter
DEBUG [PendingRangeCalculator:1] 2016-04-21 17:58:01,815
PendingRangeCalculatorService.java (line 128) No bootstrapping, leaving or
moving nodes, and no relocating tokens -> empty pending ranges for
system_traces
DEBUG [PendingRangeCalculator:1] 2016-04-21 17:58:01,815
PendingRangeCalculatorService.java (line 68) finished calculation for 6
keyspaces in 1ms
DEBUG [GossipStage:1] 2016-04-21 17:58:01,816 MigrationManager.java (line
95) Not pulling schema because versions match or shouldPullSchemaFrom
returned false
DEBUG [GossipStage:1] 2016-04-21 17:58:02,703 FailureDetector.java (line
338) Ignoring interval time of 2000848005
DEBUG [GossipStage:1] 2016-04-21 17:58:02,703 FailureDetector.java (line
338) Ignoring interval time of 2403611432
DEBUG [Background_Reporter:1] 2016-04-21 17:58:02,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [Background_Reporter:1] 2016-04-21 17:58:03,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [Background_Reporter:1] 2016-04-21 17:58:04,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [Background_Reporter:1] 2016-04-21 17:58:05,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [Background_Reporter:1] 2016-04-21 17:58:06,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [GossipStage:1] 2016-04-21 17:58:07,022 FailureDetector.java (line
338) Ignoring interval time of 2000269169
DEBUG [Background_Reporter:1] 2016-04-21 17:58:07,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [Background_Reporter:1] 2016-04-21 17:58:08,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [GossipStage:1] 2016-04-21 17:58:09,315 FailureDetector.java (line
338) Ignoring interval time of 3981962280
DEBUG [GossipStage:1] 2016-04-21 17:58:09,316 FailureDetector.java (line
338) Ignoring interval time of 2293538710
DEBUG [GossipStage:1] 2016-04-21 17:58:09,705 FailureDetector.java (line
338) Ignoring interval time of 2000566794
DEBUG [Background_Reporter:1] 2016-04-21 17:58:09,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [Background_Reporter:1] 2016-04-21 17:58:10,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [GossipStage:1] 2016-04-21 17:58:11,706 FailureDetector.java (line
338) Ignoring interval time of 2390060889
DEBUG [Background_Reporter:1] 2016-04-21 17:58:11,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [Background_Reporter:1] 2016-04-21 17:58:12,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [Background_Reporter:1] 2016-04-21 17:58:13,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [Background_Reporter:1] 2016-04-21 17:58:14,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [Background_Reporter:1] 2016-04-21 17:58:15,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [GossipStage:1] 2016-04-21 17:58:16,707 FailureDetector.java (line
338) Ignoring interval time of 2557244523
DEBUG [GossipStage:1] 2016-04-21 17:58:16,707 FailureDetector.java (line
338) Ignoring interval time of 2000746960
DEBUG [GossipStage:1] 2016-04-21 17:58:16,707 FailureDetector.java (line
338) Ignoring interval time of 2557596167
DEBUG [Background_Reporter:1] 2016-04-21 17:58:16,727 StorageService.java
(line 1401) Ignoring state change for dead or unknown endpoint: /10.0.7.4
DEBUG [ScheduledTasks:1] 2016-04-21 17:58:17,187 GCInspector.java (line
118) GC for ParNew: 140 ms for 1 collections, 2557797536 used; max is
4273995776



Please do let me know if you need anything else.

Thanks,
Mir



On Thu, Apr 21, 2016 at 10:22 AM, Anubhav Kale <Anubhav.Kale@microsoft.com>
wrote:

> Is the datastax-agent running fine on the node ? What does nodetool status
> and system.log show ?
>
>
>
> *From:* Mir Tanvir Hossain [mailto:mir.tanvir.hossain@gmail.com]
> *Sent:* Thursday, April 21, 2016 10:02 AM
> *To:* user@cassandra.apache.org
> *Subject:* Problem Replacing a Dead Node
>
>
>
> Hi, I am trying to replace a dead node with by following
> https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html
> <https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fdocs.datastax.com%2fen%2fcassandra%2f2.0%2fcassandra%2foperations%2fops_replace_node_t.html&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7ce0bd2738fe1a4640111208d36a06be83%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=kmsAE3Bd1u1mHd%2ftsJIXXAhdHMpDyxl2saXC%2fAqIi44%3d>.
> It's been 3 full days since the replacement node started, and the node is
> still not showing up as part of the cluster on OpsCenter. I was wondering
> whether the delay is due to the fact that I have a test keyspace with
> replication factor of one? If I delete that keyspace, would the new node
> successfully replace the dead node? Any general insight will be hugely
> appreciated.
>
>
>
> Thanks,
>
> Mir
>
>
>
>
>

Mime
View raw message