cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlos Rolo <r...@pythian.com>
Subject Nodes failed to bootstrap, no nodetool info but system.peer populated.
Date Mon, 11 May 2015 09:17:50 GMT
Hi all,

I just wanted to know if this should be worth filling a bug or not
(Couldn't find any similar).

I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
because of hardware failure (virtualized environment).

The process was automated, so what was supposed to happen was:

- Node 4 joins
- wait until status is UN and then 2min more
- Node 5 joins
- wait until status is UN and then 2min more
- Node 6 joins
- wait until status is UN and then 2min more

What happened:
- Node 4 joins
- Wait...
- Node 5 joins
- VM fails while node is starting.
- VM 6 starts, no node with UN, waits 2min
- Node 6 joins
- VM fails while node is starting.

After this, nodetool reports 4 nodes all UN
While trying an application (Datastax Java Driver 2.1) the debug log
reports that it tries to connect to Node 5 and 6 and fails.

Checking system.peers table, I see both nodes there. So I tried "nodetool
removenode <ID>" with the IDs in the table.

It blows up with the following exception:
Exception in thread "main" java.lang.UnsupportedOperationException: Host ID
not found.

Then I decided to do the following:
DELETE from peers where ID in (ID1, ID2);

All good, cluster still happy and driver not complaining anymore.
Is this expected behavior?



Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com

-- 


--




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message