incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jouni Hartikainen <jouni.hartikai...@reaktor.fi>
Subject Bootstrapping a new node to a virtual node cluster
Date Sat, 09 Feb 2013 12:15:47 GMT
Hello all,

I have a cluster of three nodes running 1.2.1 and I'd like to increase the capacity by adding
a new node. I'm using virtual nodes with 256 tokens and planning to use the same configuration
for the new node as well.

My cluster looks like this before adding the new node:

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns (effective)  Host ID                       
       Rack
UN  192.168.154.11    1.49 GB    256     100.0%            234b82a4-3812-4261-adab-deb805942d63
 rack1
UN  192.168.154.12    1.6 GB     256     100.0%            577db21e-81ef-45fd-a67b-cfd39455c0f6
 rack1
UN  192.168.154.13    1.64 GB    256     100.0%            6187cc5d-d44c-45cb-b738-1b87f5ae3dff
 rack1


And corresponding gossipinfo:

/192.168.154.12
  RPC_ADDRESS:192.168.154.12
  DC:datacenter1
  STATUS:NORMAL,-1072164398478041156
  LOAD:1.719425018E9
  SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
  NET_VERSION:6
  HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
  SEVERITY:0.0
  RELEASE_VERSION:1.2.1
  RACK:rack1
/192.168.154.11
  RPC_ADDRESS:192.168.154.11
  DC:datacenter1
  STATUS:NORMAL,-1158837144480089281
  LOAD:1.514343678E9
  SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
  NET_VERSION:6
  HOST_ID:234b82a4-3812-4261-adab-deb805942d63
  SEVERITY:0.0
  RELEASE_VERSION:1.2.1
  RACK:rack1
/192.168.154.13
  RPC_ADDRESS:192.168.154.13
  DC:datacenter1
  STATUS:NORMAL,-1135137292201587328
  LOAD:1.765093695E9
  SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
  NET_VERSION:6
  HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
  SEVERITY:0.0
  RELEASE_VERSION:1.2.1
  RACK:rack1
  

I have now set the correct net addresses & seeds in the cassandra.yaml of the new node
(.14) and then started it with num_tokens set to 256 and initial_token commented out. Everything
seems to go OK as I get the following prints on the log:

On node 192.168.154.11:

 INFO [GossipStage:1] 2013-02-09 12:30:28,126 Gossiper.java (line 784) Node /192.168.154.14
is now part of the cluster
 INFO [GossipStage:1] 2013-02-09 12:30:28,128 Gossiper.java (line 750) InetAddress /192.168.154.14
is now UP
 INFO [MiscStage:1] 2013-02-09 12:30:59,255 StreamOut.java (line 114) Beginning transfer to
/192.168.154.14

And on node 192.168.154.14 (the new node):

INFO 12:30:26,843 Loading persisted ring state
INFO 12:30:26,846 Starting up server gossip
WARN 12:30:26,853 No host ID found, created a4a0b918-a1c8-4acc-a050-672a96a5f110 (Note: This
should happen exactly once per node).
INFO 12:30:26,979 Starting Messaging Service on port 7000
INFO 12:30:27,014 JOINING: waiting for ring information
INFO 12:30:28,602 Node /192.168.154.11 is now part of the cluster
INFO 12:30:28,603 InetAddress /192.168.154.11 is now UP
INFO 12:30:28,675 Node /192.168.154.12 is now part of the cluster
INFO 12:30:28,678 InetAddress /192.168.154.12 is now UP
INFO 12:30:28,751 Node /192.168.154.13 is now part of the cluster
INFO 12:30:28,751 InetAddress /192.168.154.13 is now UP
INFO 12:30:29,015 JOINING: schema complete, ready to bootstrap
INFO 12:30:29,015 JOINING: getting bootstrap token
INFO 12:30:29,157 JOINING: sleeping 30000 ms for pending range setup
INFO 12:30:59,159 JOINING: Starting to bootstrap...

However, the new node does not show up in "nodetool status" (even if queried from the new
node itself):

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns (effective)  Host ID                       
       Rack
UN  192.168.154.11    1.49 GB    256     100.0%            234b82a4-3812-4261-adab-deb805942d63
 rack1
UN  192.168.154.12    1.6 GB     256     100.0%            577db21e-81ef-45fd-a67b-cfd39455c0f6
 rack1
UN  192.168.154.13    1.64 GB    256     100.0%            6187cc5d-d44c-45cb-b738-1b87f5ae3dff
 rack1

It shows up in the gossip still:

/192.168.154.12
  RPC_ADDRESS:192.168.154.12
  DC:datacenter1
  STATUS:NORMAL,-1072164398478041156
  LOAD:1.719430632E9
  SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
  NET_VERSION:6
  HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
  SEVERITY:0.0
  RELEASE_VERSION:1.2.1-SNAPSHOT
  RACK:rack1
/192.168.154.14
  RPC_ADDRESS:192.168.154.14
  DC:datacenter1
  STATUS:BOOT,8077752099299332137
  LOAD:105101.0
  SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
  NET_VERSION:6
  HOST_ID:a4a0b918-a1c8-4acc-a050-672a96a5f110
  RELEASE_VERSION:1.2.1-SNAPSHOT
  RACK:rack1
/192.168.154.11
  RPC_ADDRESS:192.168.154.11
  DC:datacenter1
  STATUS:NORMAL,-1158837144480089281
  LOAD:1.596505929E9
  SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
  NET_VERSION:6
  HOST_ID:234b82a4-3812-4261-adab-deb805942d63
  SEVERITY:0.0
  RELEASE_VERSION:1.2.1-SNAPSHOT
  RACK:rack1
/192.168.154.13
  RPC_ADDRESS:192.168.154.13
  DC:datacenter1
  STATUS:NORMAL,-1135137292201587328
  LOAD:1.765062944E9
  SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
  NET_VERSION:6
  HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
  SEVERITY:-6.776263578034403E-21
  RELEASE_VERSION:1.2.1-SNAPSHOT
  RACK:rack1


I also noticed that the new node is not listening for its thrift port (9160) until all data
streaming has ended.


Couple of questions after all this:
1) Is the new node not showing up in the "nodetool status" expected behavior?
2) Do I have to wait for all the data streaming to end until the new node starts to accept
writes? If yes, has this changed from 1.1? (I remember nodes becoming writeable immediately
after booting, but readable only after streaming has ended)
3) "nodetool info" on the new node raises an exception "Exception in thread "main" java.lang.AssertionError
at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465)" This is probably
not normal?
4) Is it so that the virtual tokens for the new node are not assigned before the data streaming
ends? ("select * from peers;" on .11 returns null for the tokens of .14) If so, how the existing
nodes decide what data to stream to a new node?


-Jouni
Mime
View raw message