cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vanger <disc...@gmail.com>
Subject Re: Adding node to Cassandra
Date Mon, 12 Mar 2012 12:46:01 GMT
Cassandra v1.0.8
once again: 4-nodes cluster, RF = 3.


On 12.03.2012 16:18, Rustam Aliyev wrote:
> What version of Cassandra do you have?
>
> On 12/03/2012 11:38, Vanger wrote:
>> We were aware of compaction overhead, but still don't understand why 
>> that shall happened: node 'D' was in stable condition, works for at 
>> least month, had all data for its token range and was comfortable 
>> with such disk space.
>> Why suddenly node needs 2x more space for data it already have? Why 
>> decreasing token range not lead to decreasing disk usage?
>>
>> On 12.03.2012 15:14, Rustam Aliyev wrote:
>>> Hi,
>>>
>>> If you use SizeTieredCompactionStrategy, you should have x2 disk 
>>> space to be on the safe side. So if you want to store 2TB data, you 
>>> need partition size of 4TB at least.  LeveledCompactionStrategy is 
>>> available in 1.x and supposed to require less free disk space (but 
>>> comes at price of I/O).
>>>
>>> --
>>> Rustam.
>>>
>>> On 12/03/2012 09:23, Vanger wrote:
>>>> *We have cassandra 4 nodes cluster* with RF = 3 (nodes named from 
>>>> 'A' to 'D', initial tokens:
>>>> *A (25%)*: 20543402371996174596346065790779111550, *
>>>> B (25%)*: 63454860067234500516210522518260948578,
>>>> *C (25%)*: 106715317233367107622067286720208938865,
>>>> *D (25%)*: 150141183460469231731687303715884105728),
>>>> *and want to add 5th node* ('E') with initial token = 
>>>> 164163260474281062972548100673162157075,  then we want to rebalance 
>>>> A, D, E nodes such way they'll own equal percentage of data. All 
>>>> nodes have ~400 GB of data and around ~300GB disk free space.
>>>> What we did:
>>>> 1. 'Join' new cassandra instance (node 'E') to cluster and wait 
>>>> 'till it loads data for it tokens range.
>>>>
>>>> 2. Move node 'D' initial token down from 150... to 130...
>>>> Here we ran into a problem. When "move" started disk usage for node 
>>>> C grows from 400 to 750GB, we saw running compactions on node 'D' 
>>>> but some compactions failed with /"WARN [CompactionExecutor:580] 
>>>> 2012-03-11 16:57:56,036 CompactionTask.java (line 87) insufficient 
>>>> space to compact all requested files SSTableReader"/ after that we 
>>>> killed "move" process to avoid "out of disk space" error (when 5GB 
>>>> of free space left). After restart it frees 100GB of space and now 
>>>> we have total of 105GB free disk space on node 'D'. Also we noticed 
>>>> increased disk usage by ~150GB at node 'B' but it stops growing 
>>>> before we stopped "move token".
>>>>
>>>>
>>>> So now we have 5 nodes in cluster in status like this:
>>>> Node, Owns%,     Load,     Init. token
>>>> A:         16%       400GB        020...
>>>> B:         25%       520GB        063...
>>>> C:         25%       400GB        106...
>>>> D:         25%       640GB        150...
>>>> E:          9%         300GB        164...
>>>>
>>>> We'll add disk space for all nodes and run some cleanups, but 
>>>> there's still left some questions:
>>>>
>>>> What is the best next step  for us from this point?
>>>> What is correct procedure after all and what should we expect when 
>>>> adding node to cassandra cluster?
>>>> We expected decrease of used disk space on node 'D' 'cause we 
>>>> shrink token range for this node, but saw the opposite, why it 
>>>> happened and is it normal behavior?
>>>> What if we'll have 2TB of data on 2.5TB disk and we wanted to add 
>>>> another node and move tokens?
>>>> Is it possible to automate node addition to cluster and be sure we 
>>>> won't run out of space?
>>>>
>>>> Thank.
>>


Mime
View raw message