cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 土卜皿 <pengcz.n...@gmail.com>
Subject Re: why one of the new added nodes' bootstrap is very slow?
Date Thu, 28 Jan 2016 01:57:15 GMT
Hi Alain,
Thank you very much!


> UJ  192.21.0.185  299.22 GB  256     ?       84c0dd16-6491-4bfb-b288-d4e410cd8c2a  RAC1
>> UN  192.21.0.184  670.14 MB  256     ?       4041c232-c110-4315-89a1-23ca53b851c2
 RAC1
>>
>>
> Obviously .184 didn't bootstrap correctly. When a node is added, it
> becomes responsible for a range (multiple ranges with vnodes), so it has to
> receive data from nodes previously responsible for this (these) range(s).
> So 600 MB looks wrong.
>

What should I do for this wrong bootstrap?


>
> So .185 is behaving as expected, .184 isn't.
>
> Yet .185 having twice the data from other node is weird unless you changed
> Replication factor or streamed data multiple time (then compaction will
> eventually fix this).
>

No, I did not change Replication factor


> Plus this node has less tokens than the first 3 nodes.
> Are you running heterogeneous hardware ?
>

Yes, the old nodes with Memory: 64G, Disk: 4 X 1.1T and CPU: 16 cores, the
old nodes with: Memory 32G, Disk: 1 X 460G and CPU: 32 cores



> Why setting 512 token for the 3 first nodes, and 256 for other nodes ?
> From what I heard default vnodes is a way too high, you generally want to
> go with something between 16 and 64 on production (if it is not too late).
>

No why :-), the 512 is from some one example, 256 because I used different
hardware, I can modified all the numbers after I add these new nodes
successfully?


So I restarted it and the join continued! I don't know why there is the
>> difference between the two nodes?
>>
> My guess is the join did not continue. Once you bootstrap a node, system
> keyspace is filled up with some information. If the bootstrap fails, you
> need to wipe the data directory. I advice you to directly "rm -rf
> /path_to_cassandra/data/*".
>
> If you don't remove system KS, node will behave as he is already part of
> the ring and so, won't stream anything, it won't bootstrap, just start. So
> that would be the difference imho.
>
> If you just wipe the system keyspace (not your data), it will work, yet
> you will end up streaming the same data and will need to compact, adding
> useless work.
>
> So I would go clean stat and start the process again.
>
Sorry, I am not so clear for the above description, you mean:

Under "192.21.0.185 229.2GB", I can directly "rm -rf
/path_to_cassandra/data/" without changing anything else, and start the
cassadra again?

Under "192.21.0.184 670.14MB",  I would do something  as you said "So I
would go clean stat and start the process again.",  what commands I should
use to do it?

Thank you very much!


Best REGARDS
Dillon


>
> https://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureDataDistributeVnodesUsing_c.html
>
> I would advise you to read documentation on datastax website, it will save
> you a lot of time and troubles imho. Even if I am glad to help.
>
> C*heers,
>
> -----------------
> Alain Rodriguez
> France
>
> The Last Pickle
> http://www.thelastpickle.com
>
> 2016-01-27 14:11 GMT+01:00 土卜皿 <pengcz.nwpu@gmail.com>:
>
>> Hi
>>
>> Cassandra version: 2.1.11
>>
>> The existed cluster has three nodes:
>>
>> [root@report-02 cassandra]# bin/nodetool status
>> UN  192.21.0.135  120.85 GB  512     ?       11e1e80f-9c5f-4f7c-81f2-42d3b704d8e3
 RAC1
>> UN  192.21.0.133  129.13 GB  512     ?       3e662ccb-fa2b-427b-9ca1-c2d3468bfbc9
 RAC1
>> UN  192.21.0.131  149.05 GB  512     ?       60f763f3-09bc-4d6f-9301-494c93857fc1
 RAC1
>>
>> I wanted to add two nodes and set the same configs as the cluster's nodes.
>>
>> node1: 192.21.0.184
>> node2: 192.21.0.185
>>
>> After starting the two nodes one by one, the first node 192.21.0.184 finished
>> the joining immediately, but the second one 192.21.0.185 took more than
>> 24 hours to join and not finished now:
>>
>> Under 192.168.0.184:
>>
>> [root@report-01 cassandra]# bin/nodetool compactionstats
>> pending tasks: 0
>>
>> Under 192.168.0.185:
>>
>>  [root@report-02 cassandra]# bin/nodetool compactionstats
>>  pending tasks: 21
>>  compaction type      keyspace       table     completed          total    unit 
 progress
>>  Compaction   testforuser   users1027    6204396079    14923537640   bytes     41.57%
>>  Compaction   user_center       users   19325435997   514143044706   bytes      3.76%
>>  Compaction   user_center       users   12305639479   118703090319   bytes     10.37%
>>  Active compaction remaining time :  10h05m54s
>>
>> And:
>>
>> [root@report-02 cassandra]# bin/nodetool status
>> Datacenter: DC1
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address       Load       Tokens  Owns    Host ID                            
  Rack
>> UN  192.21.0.135  120.85 GB  512     ?       11e1e80f-9c5f-4f7c-81f2-42d3b704d8e3
 RAC1
>> UN  192.21.0.133  129.13 GB  512     ?       3e662ccb-fa2b-427b-9ca1-c2d3468bfbc9
 RAC1
>> UN  192.21.0.131  149.05 GB  512     ?       60f763f3-09bc-4d6f-9301-494c93857fc1
 RAC1
>> UJ  192.21.0.185  299.22 GB  256     ?       84c0dd16-6491-4bfb-b288-d4e410cd8c2a
 RAC1
>> UN  192.21.0.184  670.14 MB  256     ?       4041c232-c110-4315-89a1-23ca53b851c2
 RAC1
>>
>> From the above load data size, obviously, node2(192.21.0.185)'s 299.22G
>> is not normal.
>>
>> And the node2's boostrap interrupted several times because it got a error:
>>
>> INFO  00:57:42 [Stream #8eb8cbe0-c488-11e5-baf9-918c8558de90] Session with /192.21.0.135
is complete
>> INFO  00:57:42 [Stream #8eb8cbe0-c488-11e5-baf9-918c8558de90] Session with /192.21.0.131
is complete
>> WARN  00:57:42 [Stream #8eb8cbe0-c488-11e5-baf9-918c8558de90] Stream failed
>> ERROR 00:57:42 Exception encountered during startup
>> java.lang.RuntimeException: Error during boostrap: Stream failed
>>
>> So I restarted it and the join continued!
>>
>> I don't know why there is the difference between the two nodes?
>>
>> I should stop it, and change something?
>>
>> Thank you in advance!
>>
>> Dillon
>>
>
>

Mime
View raw message