cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mariusz Dymarek <mdyma...@opera.com>
Subject Re: Tripling size of a cluster
Date Fri, 20 Jul 2012 10:22:24 GMT
On 20.07.2012 11:02, aaron morton wrote:
> I would check for stored hints in /var/lib/cassandra/data/system
>
hmm where i can find this kind of info?
i can see HintsColumnFamily cf inside system kespace, but it`s empty...
> Putting nodes in different racks can make placement tricky so…
> Are you running a multi DC setup ?
No we have all nodes in one Data Center(all entries in 
cassandra-topology.properties file contain same DC name
Are you using the NTS ?
no we`re using:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
> What is the RF setting ?
Options: [replication_factor:2]
  What setting do you have for the Snitch ?
endpoint_snitch: org.apache.cassandra.locator.PropertyFileSnitch
> What is the full node assignments.
Address   DC    Rack    Status State   Load    Effective-Owership  Token
node01-01   iponly   rack1   Up   Normal  255.82 GB   6.67%   0
node02-07   iponly   rack2   Up   Normal  255.05 GB   6.67% 
5671372782015641057722910123862803524
node01-07   iponly   rack1   Up   Normal  254.65 GB   6.67% 
11342745564031282115445820247725607048
node02-01   iponly   rack2   Up   Normal  261.2 GB    6.67% 
17014118346046923173168730371588410572
node01-08   iponly   rack1   Up   Normal  254.08 GB   6.67% 
22685491128062564230891640495451214097
node02-08   iponly   rack2   Up   Normal  252.04 GB   6.67% 
28356863910078205288614550619314017621
node01-02   iponly   rack1   Up   Normal  256.44 GB   6.67% 
34028236692093846346337460743176821144
node02-09   iponly   rack2   Up   Normal  255.09 GB   6.67% 
39699609474109487404060370867039624669
node01-09   iponly   rack1   Up   Normal  254.28 GB   6.67% 
45370982256125128461783280990902428194
node02-02   iponly   rack2   Up   Normal  259.54 GB   6.67% 
51042355038140769519506191114765231716
node01-010   iponly   rack1   Up   Normal  258.43 GB   6.67% 
56713727820156410577229101238628035242
node02-010   iponly   rack2   Up   Normal  258.09 GB   6.67% 
62385100602172051634952011362490838766
node01-03   iponly   rack1   Up   Normal  253.35 GB   6.67% 
68056473384187692692674921486353642288
node02-11   iponly   rack2   Up   Normal  256.47 GB   6.67% 
73727846166203333750397831610216445815
node01-11   iponly   rack1   Up   Normal  255.49 GB   6.67% 
79399218948218974808120741734079249339
node02-03   iponly   rack2   Up   Normal  252.7 GB    6.67% 
85070591730234615865843651857942052860
node01-012   iponly   rack1   Up   Normal  258.64 GB   6.67% 
90741964512250256923566561981804856388
node02-012   iponly   rack2   Up   Normal  257.88 GB   6.67% 
96413337294265897981289472105667659912
node01-05   iponly   rack1   Up   Normal  254.9 GB    6.67% 
102084710076281539039012382229530463432
node02-013   iponly   rack2   Up   Normal  251.29 GB   6.67% 
107756082858297180096735292353393266961
node01-013   iponly   rack1   Up   Normal  253.99 GB   6.67% 
113427455640312821154458202477256070485
node02-05   iponly   rack2   Up   Normal  252.96 GB   6.67% 
119098828422328462212181112601118874004
node01-14   iponly   rack1   Up   Normal  255.13 GB   6.67% 
124770201204344103269904022724981677533
node02-14   iponly   rack2   Up   Normal  404.65 GB   6.67% 
130441573986359744327626932848844481058
node01-06   iponly   rack1   Up   Normal  257.56 GB   6.67% 
136112946768375385385349842972707284576
node02-015   iponly   rack2   Up   Normal  297.18 GB   6.67% 
141784319550391026443072753096570088106
node01-15   iponly   rack1   Up   Normal  254.95 GB   6.67% 
147455692332406667500795663220432891630
node02-06   iponly   rack2   Up   Normal  255.97 GB   6.67% 
153127065114422308558518573344295695148
node01-16   iponly   rack1   Up   Normal  256.17 GB   6.67% 
158798437896437949616241483468158498679
node02-16   iponly   rack2   Up   Normal  257.17 GB   6.67% 
164469810678453590673964393592021302203

>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 19/07/2012, at 6:00 PM, Mariusz Dymarek wrote:
>
>> Hi again,
>> we have now moved all nodes to correct position in ring, but we can
>> see higher load on 2 nodes, than on other nodes:
>> ...
>> node01-05 rack1 Up Normal 244.65 GB 6,67%
>> 102084710076281539039012382229530463432
>> node02-13 rack2 Up Normal 240.26 GB 6,67%
>> 107756082858297180096735292353393266961
>> node01-13 rack1 Up Normal 243.75 GB 6,67%
>> 113427455640312821154458202477256070485
>> node02-05 rack2 Up Normal 249.31 GB 6,67%
>> 119098828422328462212181112601118874004
>> node01-14 rack1 Up Normal 244.95 GB 6,67%
>> 124770201204344103269904022724981677533
>> node02-14 rack2 Up Normal 392.7 GB 6,67%
>> 130441573986359744327626932848844481058
>> node01-06 rack1 Up Normal 249.3 GB 6,67%
>> 136112946768375385385349842972707284576
>> node02-15 rack2 Up Normal 286.82 GB 6,67%
>> 141784319550391026443072753096570088106
>> node01-15 rack1 Up Normal 245.21 GB 6,67%
>> 147455692332406667500795663220432891630
>> node02-06 rack2 Up Normal 244.9 GB 6,67%
>> 153127065114422308558518573344295695148
>> ...
>>
>> Node:
>> * node01-15 = > 286.82 GB
>> * node02-14 = > 392.7 GB
>>
>> average load on all other nodes is around 245 GB, nodetool cleanup
>> command was invoked on problematic nodes after move operation...
>> Why this has happen?
>> And how can we balance cluster?
>> On 06.07.2012 20:15, aaron morton wrote:
>>> If you have the time yes I would wait for the bootstrap to finish. It
>>> will make you life easier.
>>>
>>> good luck.
>>>
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 6/07/2012, at 7:12 PM, Mariusz Dymarek wrote:
>>>
>>>> Hi,
>>>> we`re in the middle of extending our cluster from 10 to 30 nodes,
>>>> we`re running cassandra 1.1.1...
>>>> We`ve generated initial tokens for new nodes:
>>>> "0": 0, # existing: node01-01
>>>> "1": 5671372782015641057722910123862803524, # new: node02-07
>>>> "2": 11342745564031282115445820247725607048, # new: node01-07
>>>> "3": 17014118346046923173168730371588410572, # existing: node02-01
>>>> "4": 22685491128062564230891640495451214097, # new: node01-08
>>>> "5": 28356863910078205288614550619314017621, # new: node02-08
>>>> "6": 34028236692093846346337460743176821145, # existing: node01-02
>>>> "7": 39699609474109487404060370867039624669, # new: node02-09
>>>> "8": 45370982256125128461783280990902428194, # new: node01-09
>>>> "9": 51042355038140769519506191114765231718, # existing: node02-02
>>>> "10": 56713727820156410577229101238628035242, # new: node01-10
>>>> "11": 62385100602172051634952011362490838766, # new: node02-10
>>>> "12": 68056473384187692692674921486353642291, # existing: node01-03
>>>> "13": 73727846166203333750397831610216445815, # new: node02-11
>>>> "14": 79399218948218974808120741734079249339, # new: node01-11
>>>> "15": 85070591730234615865843651857942052864, # existing: node02-03
>>>> "16": 90741964512250256923566561981804856388, # new: node01-12
>>>> "17": 96413337294265897981289472105667659912, # new: node02-12
>>>> "18": 102084710076281539039012382229530463436, # existing: node01-05
>>>> "19": 107756082858297180096735292353393266961, # new: node02-13
>>>> "20": 113427455640312821154458202477256070485, # new: node01-13
>>>> "21": 119098828422328462212181112601118874009, # existing: node02-05
>>>> "22": 124770201204344103269904022724981677533, # new: node01-14
>>>> "23": 130441573986359744327626932848844481058, # new: node02-14
>>>> "24": 136112946768375385385349842972707284582, # existing: node01-06
>>>> "25": 141784319550391026443072753096570088106, # new: node02-15
>>>> "26": 147455692332406667500795663220432891630, # new: node01-15
>>>> "27": 153127065114422308558518573344295695155, # existing: node02-06
>>>> "28": 158798437896437949616241483468158498679, # new: node01-16
>>>> "29": 164469810678453590673964393592021302203 # new: node02-16
>>>> then we`ve started to boostrap new nodes,
>>>> but due to copy and paste mistake:
>>>> * node node01-14 was started with
>>>> 130441573986359744327626932848844481058 as initial token(so node01-14
>>>> has initial_token, what should belong to node02-14), it
>>>> should have 124770201204344103269904022724981677533 as initial_token
>>>> * node node02-14 was started with
>>>> 136112946768375385385349842972707284582 as initial token, so it has
>>>> token from existing node01-06....
>>>>
>>>> However we`ve used other program for generating previous
>>>> initial_tokens and actual token of node01-06 in ring is
>>>> 136112946768375385385349842972707284576.
>>>> Summing up: we have currently this situation in ring:
>>>>
>>>> node02-05 rack2 Up Normal 596.31 GB 6.67%
>>>> 119098828422328462212181112601118874004
>>>> node01-14 rack1 Up Joining 242.92 KB 0.00%
>>>> 130441573986359744327626932848844481058
>>>> node01-06 rack1 Up Normal 585.5 GB 13.33%
>>>> 136112946768375385385349842972707284576
>>>> node02-14 rack2 Up Joining 113.17 KB 0.00%
>>>> 136112946768375385385349842972707284582
>>>> node02-15 rack2 Up Joining 178.05 KB 0.00%
>>>> 141784319550391026443072753096570088106
>>>> node01-15 rack1 Up Joining 191.7 GB 0.00%
>>>> 147455692332406667500795663220432891630
>>>> node02-06 rack2 Up Normal 597.69 GB 20.00%
>>>> 153127065114422308558518573344295695148
>>>>
>>>>
>>>> We would like to get back to our original configuration.
>>>> Is it safe to wait for finishing bootstraping of all new nodes and
>>>> after that invoke:
>>>> * nodetool -h node01-14 move 124770201204344103269904022724981677533
>>>> * nodetool -h node02-14 move 130441573986359744327626932848844481058
>>>> We should probably run nodetool cleanup on several nodes after that...
>>>> Regards
>>>> Dymarek Mariusz
>>>
>>
>


Mime
View raw message