Hello all,
I have a cluster of three nodes running 1.2.1 and I'd like to increase the capacity by adding
a new node. I'm using virtual nodes with 256 tokens and planning to use the same configuration
for the new node as well.
My cluster looks like this before adding the new node:
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 192.168.154.11 1.49 GB 256 100.0% 234b82a4-3812-4261-adab-deb805942d63
rack1
UN 192.168.154.12 1.6 GB 256 100.0% 577db21e-81ef-45fd-a67b-cfd39455c0f6
rack1
UN 192.168.154.13 1.64 GB 256 100.0% 6187cc5d-d44c-45cb-b738-1b87f5ae3dff
rack1
And corresponding gossipinfo:
/192.168.154.12
RPC_ADDRESS:192.168.154.12
DC:datacenter1
STATUS:NORMAL,-1072164398478041156
LOAD:1.719425018E9
SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
NET_VERSION:6
HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
SEVERITY:0.0
RELEASE_VERSION:1.2.1
RACK:rack1
/192.168.154.11
RPC_ADDRESS:192.168.154.11
DC:datacenter1
STATUS:NORMAL,-1158837144480089281
LOAD:1.514343678E9
SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
NET_VERSION:6
HOST_ID:234b82a4-3812-4261-adab-deb805942d63
SEVERITY:0.0
RELEASE_VERSION:1.2.1
RACK:rack1
/192.168.154.13
RPC_ADDRESS:192.168.154.13
DC:datacenter1
STATUS:NORMAL,-1135137292201587328
LOAD:1.765093695E9
SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
NET_VERSION:6
HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
SEVERITY:0.0
RELEASE_VERSION:1.2.1
RACK:rack1
I have now set the correct net addresses & seeds in the cassandra.yaml of the new node
(.14) and then started it with num_tokens set to 256 and initial_token commented out. Everything
seems to go OK as I get the following prints on the log:
On node 192.168.154.11:
INFO [GossipStage:1] 2013-02-09 12:30:28,126 Gossiper.java (line 784) Node /192.168.154.14
is now part of the cluster
INFO [GossipStage:1] 2013-02-09 12:30:28,128 Gossiper.java (line 750) InetAddress /192.168.154.14
is now UP
INFO [MiscStage:1] 2013-02-09 12:30:59,255 StreamOut.java (line 114) Beginning transfer to
/192.168.154.14
And on node 192.168.154.14 (the new node):
INFO 12:30:26,843 Loading persisted ring state
INFO 12:30:26,846 Starting up server gossip
WARN 12:30:26,853 No host ID found, created a4a0b918-a1c8-4acc-a050-672a96a5f110 (Note: This
should happen exactly once per node).
INFO 12:30:26,979 Starting Messaging Service on port 7000
INFO 12:30:27,014 JOINING: waiting for ring information
INFO 12:30:28,602 Node /192.168.154.11 is now part of the cluster
INFO 12:30:28,603 InetAddress /192.168.154.11 is now UP
INFO 12:30:28,675 Node /192.168.154.12 is now part of the cluster
INFO 12:30:28,678 InetAddress /192.168.154.12 is now UP
INFO 12:30:28,751 Node /192.168.154.13 is now part of the cluster
INFO 12:30:28,751 InetAddress /192.168.154.13 is now UP
INFO 12:30:29,015 JOINING: schema complete, ready to bootstrap
INFO 12:30:29,015 JOINING: getting bootstrap token
INFO 12:30:29,157 JOINING: sleeping 30000 ms for pending range setup
INFO 12:30:59,159 JOINING: Starting to bootstrap...
However, the new node does not show up in "nodetool status" (even if queried from the new
node itself):
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 192.168.154.11 1.49 GB 256 100.0% 234b82a4-3812-4261-adab-deb805942d63
rack1
UN 192.168.154.12 1.6 GB 256 100.0% 577db21e-81ef-45fd-a67b-cfd39455c0f6
rack1
UN 192.168.154.13 1.64 GB 256 100.0% 6187cc5d-d44c-45cb-b738-1b87f5ae3dff
rack1
It shows up in the gossip still:
/192.168.154.12
RPC_ADDRESS:192.168.154.12
DC:datacenter1
STATUS:NORMAL,-1072164398478041156
LOAD:1.719430632E9
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
SEVERITY:0.0
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
/192.168.154.14
RPC_ADDRESS:192.168.154.14
DC:datacenter1
STATUS:BOOT,8077752099299332137
LOAD:105101.0
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:a4a0b918-a1c8-4acc-a050-672a96a5f110
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
/192.168.154.11
RPC_ADDRESS:192.168.154.11
DC:datacenter1
STATUS:NORMAL,-1158837144480089281
LOAD:1.596505929E9
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:234b82a4-3812-4261-adab-deb805942d63
SEVERITY:0.0
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
/192.168.154.13
RPC_ADDRESS:192.168.154.13
DC:datacenter1
STATUS:NORMAL,-1135137292201587328
LOAD:1.765062944E9
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
SEVERITY:-6.776263578034403E-21
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
I also noticed that the new node is not listening for its thrift port (9160) until all data
streaming has ended.
Couple of questions after all this:
1) Is the new node not showing up in the "nodetool status" expected behavior?
2) Do I have to wait for all the data streaming to end until the new node starts to accept
writes? If yes, has this changed from 1.1? (I remember nodes becoming writeable immediately
after booting, but readable only after streaming has ended)
3) "nodetool info" on the new node raises an exception "Exception in thread "main" java.lang.AssertionError
at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465)" This is probably
not normal?
4) Is it so that the virtual tokens for the new node are not assigned before the data streaming
ends? ("select * from peers;" on .11 returns null for the tokens of .14) If so, how the existing
nodes decide what data to stream to a new node?
-Jouni
|