incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Bootstrapping a new node to a virtual node cluster
Date Tue, 12 Feb 2013 16:58:27 GMT
Just checking if this sorted it's self out? 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 10/02/2013, at 1:15 AM, Jouni Hartikainen <jouni.hartikainen@reaktor.fi> wrote:

> Hello all,
> 
> I have a cluster of three nodes running 1.2.1 and I'd like to increase the capacity by
adding a new node. I'm using virtual nodes with 256 tokens and planning to use the same configuration
for the new node as well.
> 
> My cluster looks like this before adding the new node:
> 
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns (effective)  Host ID                  
            Rack
> UN  192.168.154.11    1.49 GB    256     100.0%            234b82a4-3812-4261-adab-deb805942d63
 rack1
> UN  192.168.154.12    1.6 GB     256     100.0%            577db21e-81ef-45fd-a67b-cfd39455c0f6
 rack1
> UN  192.168.154.13    1.64 GB    256     100.0%            6187cc5d-d44c-45cb-b738-1b87f5ae3dff
 rack1
> 
> 
> And corresponding gossipinfo:
> 
> /192.168.154.12
>  RPC_ADDRESS:192.168.154.12
>  DC:datacenter1
>  STATUS:NORMAL,-1072164398478041156
>  LOAD:1.719425018E9
>  SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
>  NET_VERSION:6
>  HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
>  SEVERITY:0.0
>  RELEASE_VERSION:1.2.1
>  RACK:rack1
> /192.168.154.11
>  RPC_ADDRESS:192.168.154.11
>  DC:datacenter1
>  STATUS:NORMAL,-1158837144480089281
>  LOAD:1.514343678E9
>  SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
>  NET_VERSION:6
>  HOST_ID:234b82a4-3812-4261-adab-deb805942d63
>  SEVERITY:0.0
>  RELEASE_VERSION:1.2.1
>  RACK:rack1
> /192.168.154.13
>  RPC_ADDRESS:192.168.154.13
>  DC:datacenter1
>  STATUS:NORMAL,-1135137292201587328
>  LOAD:1.765093695E9
>  SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
>  NET_VERSION:6
>  HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
>  SEVERITY:0.0
>  RELEASE_VERSION:1.2.1
>  RACK:rack1
> 
> 
> I have now set the correct net addresses & seeds in the cassandra.yaml of the new
node (.14) and then started it with num_tokens set to 256 and initial_token commented out.
Everything seems to go OK as I get the following prints on the log:
> 
> On node 192.168.154.11:
> 
> INFO [GossipStage:1] 2013-02-09 12:30:28,126 Gossiper.java (line 784) Node /192.168.154.14
is now part of the cluster
> INFO [GossipStage:1] 2013-02-09 12:30:28,128 Gossiper.java (line 750) InetAddress /192.168.154.14
is now UP
> INFO [MiscStage:1] 2013-02-09 12:30:59,255 StreamOut.java (line 114) Beginning transfer
to /192.168.154.14
> 
> And on node 192.168.154.14 (the new node):
> 
> INFO 12:30:26,843 Loading persisted ring state
> INFO 12:30:26,846 Starting up server gossip
> WARN 12:30:26,853 No host ID found, created a4a0b918-a1c8-4acc-a050-672a96a5f110 (Note:
This should happen exactly once per node).
> INFO 12:30:26,979 Starting Messaging Service on port 7000
> INFO 12:30:27,014 JOINING: waiting for ring information
> INFO 12:30:28,602 Node /192.168.154.11 is now part of the cluster
> INFO 12:30:28,603 InetAddress /192.168.154.11 is now UP
> INFO 12:30:28,675 Node /192.168.154.12 is now part of the cluster
> INFO 12:30:28,678 InetAddress /192.168.154.12 is now UP
> INFO 12:30:28,751 Node /192.168.154.13 is now part of the cluster
> INFO 12:30:28,751 InetAddress /192.168.154.13 is now UP
> INFO 12:30:29,015 JOINING: schema complete, ready to bootstrap
> INFO 12:30:29,015 JOINING: getting bootstrap token
> INFO 12:30:29,157 JOINING: sleeping 30000 ms for pending range setup
> INFO 12:30:59,159 JOINING: Starting to bootstrap...
> 
> However, the new node does not show up in "nodetool status" (even if queried from the
new node itself):
> 
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns (effective)  Host ID                  
            Rack
> UN  192.168.154.11    1.49 GB    256     100.0%            234b82a4-3812-4261-adab-deb805942d63
 rack1
> UN  192.168.154.12    1.6 GB     256     100.0%            577db21e-81ef-45fd-a67b-cfd39455c0f6
 rack1
> UN  192.168.154.13    1.64 GB    256     100.0%            6187cc5d-d44c-45cb-b738-1b87f5ae3dff
 rack1
> 
> It shows up in the gossip still:
> 
> /192.168.154.12
>  RPC_ADDRESS:192.168.154.12
>  DC:datacenter1
>  STATUS:NORMAL,-1072164398478041156
>  LOAD:1.719430632E9
>  SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
>  NET_VERSION:6
>  HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
>  SEVERITY:0.0
>  RELEASE_VERSION:1.2.1-SNAPSHOT
>  RACK:rack1
> /192.168.154.14
>  RPC_ADDRESS:192.168.154.14
>  DC:datacenter1
>  STATUS:BOOT,8077752099299332137
>  LOAD:105101.0
>  SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
>  NET_VERSION:6
>  HOST_ID:a4a0b918-a1c8-4acc-a050-672a96a5f110
>  RELEASE_VERSION:1.2.1-SNAPSHOT
>  RACK:rack1
> /192.168.154.11
>  RPC_ADDRESS:192.168.154.11
>  DC:datacenter1
>  STATUS:NORMAL,-1158837144480089281
>  LOAD:1.596505929E9
>  SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
>  NET_VERSION:6
>  HOST_ID:234b82a4-3812-4261-adab-deb805942d63
>  SEVERITY:0.0
>  RELEASE_VERSION:1.2.1-SNAPSHOT
>  RACK:rack1
> /192.168.154.13
>  RPC_ADDRESS:192.168.154.13
>  DC:datacenter1
>  STATUS:NORMAL,-1135137292201587328
>  LOAD:1.765062944E9
>  SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
>  NET_VERSION:6
>  HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
>  SEVERITY:-6.776263578034403E-21
>  RELEASE_VERSION:1.2.1-SNAPSHOT
>  RACK:rack1
> 
> 
> I also noticed that the new node is not listening for its thrift port (9160) until all
data streaming has ended.
> 
> 
> Couple of questions after all this:
> 1) Is the new node not showing up in the "nodetool status" expected behavior?
> 2) Do I have to wait for all the data streaming to end until the new node starts to accept
writes? If yes, has this changed from 1.1? (I remember nodes becoming writeable immediately
after booting, but readable only after streaming has ended)
> 3) "nodetool info" on the new node raises an exception "Exception in thread "main" java.lang.AssertionError
at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465)" This is probably
not normal?
> 4) Is it so that the virtual tokens for the new node are not assigned before the data
streaming ends? ("select * from peers;" on .11 returns null for the tokens of .14) If so,
how the existing nodes decide what data to stream to a new node?
> 
> 
> -Jouni


Mime
View raw message