Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <4F5DDA90.6070002@code.az>
Date: Mon, 12 Mar 2012 11:14:24 +0000
From: Rustam Aliyev <rustam@code.az>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
 rv:10.0.1) Gecko/20120208 Thunderbird/10.0.1
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Re: Adding node to Cassandra
References: <4F5DC081.5070500@gmail.com>
In-Reply-To: <4F5DC081.5070500@gmail.com>
Content-Type: multipart/alternative;
 boundary="------------030906040704060604070708"

This is a multi-part message in MIME format.
--------------030906040704060604070708
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

If you use SizeTieredCompactionStrategy, you should have x2 disk space 
to be on the safe side. So if you want to store 2TB data, you need 
partition size of 4TB at least.  LeveledCompactionStrategy is available 
in 1.x and supposed to require less free disk space (but comes at price 
of I/O).

--
Rustam.

On 12/03/2012 09:23, Vanger wrote:
> *We have cassandra 4 nodes cluster* with RF = 3 (nodes named from 'A' 
> to 'D', initial tokens:
> *A (25%)*: 20543402371996174596346065790779111550, *
> B (25%)*: 63454860067234500516210522518260948578,
> *C (25%)*: 106715317233367107622067286720208938865,
> *D (25%)*: 150141183460469231731687303715884105728),
> *and want to add 5th node* ('E') with initial token = 
> 164163260474281062972548100673162157075,  then we want to rebalance A, 
> D, E nodes such way they'll own equal percentage of data. All nodes 
> have ~400 GB of data and around ~300GB disk free space.
> What we did:
> 1. 'Join' new cassandra instance (node 'E') to cluster and wait 'till 
> it loads data for it tokens range.
>
> 2. Move node 'D' initial token down from 150... to 130...
> Here we ran into a problem. When "move" started disk usage for node C 
> grows from 400 to 750GB, we saw running compactions on node 'D' but 
> some compactions failed with /"WARN [CompactionExecutor:580] 
> 2012-03-11 16:57:56,036 CompactionTask.java (line 87) insufficient 
> space to compact all requested files SSTableReader"/ after that we 
> killed "move" process to avoid "out of disk space" error (when 5GB of 
> free space left). After restart it frees 100GB of space and now we 
> have total of 105GB free disk space on node 'D'. Also we noticed 
> increased disk usage by ~150GB at node 'B' but it stops growing before 
> we stopped "move token".
>
>
> So now we have 5 nodes in cluster in status like this:
> Node, Owns%,     Load,     Init. token
> A:         16%       400GB        020...
> B:         25%       520GB        063...
> C:         25%       400GB        106...
> D:         25%       640GB        150...
> E:          9%         300GB        164...
>
> We'll add disk space for all nodes and run some cleanups, but there's 
> still left some questions:
>
> What is the best next step  for us from this point?
> What is correct procedure after all and what should we expect when 
> adding node to cassandra cluster?
> We expected decrease of used disk space on node 'D' 'cause we shrink 
> token range for this node, but saw the opposite, why it happened and 
> is it normal behavior?
> What if we'll have 2TB of data on 2.5TB disk and we wanted to add 
> another node and move tokens?
> Is it possible to automate node addition to cluster and be sure we 
> won't run out of space?
>
> Thank.

--------------030906040704060604070708
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <font size="-1"><font face="Helvetica, Arial, sans-serif">Hi,<br>
        <br>
        If you use SizeTieredCompactionStrategy, you should have x2 disk
        space to</font></font> be on the safe side. So if you want to
    store 2TB data, you need partition size of 4TB at least. 
    LeveledCompactionStrategy is available in 1.x and supposed to
    require less free disk space (but comes at price of I/O).<br>
    <br>
    --<br>
    Rustam.<br>
    <br>
    On 12/03/2012 09:23, Vanger wrote:
    <blockquote cite="mid:4F5DC081.5070500@gmail.com" type="cite">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <b>We have cassandra 4 nodes cluster</b> with RF = 3 (nodes named
      from 'A' to 'D', initial tokens:     <br>
      <b>A (25%)</b>: 20543402371996174596346065790779111550,     <b><br>
        B (25%)</b>: 63454860067234500516210522518260948578,     <br>
      <b>C (25%)</b>: 106715317233367107622067286720208938865,    <br>
      <b>D (25%)</b>: 150141183460469231731687303715884105728), <br>
      <b>and want to add 5th node</b> ('E') with initial token =
      164163260474281062972548100673162157075,  then we want to
      rebalance A, D, E nodes such way they'll own equal percentage of
      data. All nodes have ~400 GB of data and around ~300GB disk free
      space.<br>
      What we did:<br>
      1. 'Join' new cassandra instance (node 'E') to cluster and wait
      'till it loads data for it tokens range.<br>
      <br>
      2. Move node 'D' initial token down from 150... to 130... <br>
      Here we ran into a problem. When "move" started disk usage for
      node C grows from 400 to 750GB, we saw running compactions on node
      'D' but some compactions failed with <i>"WARN
        [CompactionExecutor:580] 2012-03-11 16:57:56,036
        CompactionTask.java (line 87) insufficient space to compact all
        requested files SSTableReader"</i> after that we killed "move"
      process to avoid "out of disk space" error (when 5GB of free space
      left). After restart it frees 100GB of space and now we have total
      of 105GB free disk space on node 'D'. Also we noticed increased
      disk usage by ~150GB at node 'B' but it stops growing before we
      stopped "move token".<br>
      <br>
      <br>
      So now we have 5 nodes in cluster in status like this:<br>
      Node, Owns%,     Load,     Init. token<br>
      A:         16%       400GB        020...<br>
      B:         25%       520GB        063...<br>
      C:         25%       400GB        106...<br>
      D:         25%       640GB        150...<br>
      E:          9%         300GB        164...<br>
      <br>
      We'll add disk space for all nodes and run some cleanups, but
      there's still left some questions:<br>
      <br>
      What is the best next step  for us from this point?<br>
      What is correct procedure after all and what should we expect when
      adding node to cassandra cluster?<br>
      We expected decrease of used disk space on node 'D' 'cause we
      shrink token range for this node, but saw the opposite, why it
      happened and is it normal behavior?<br>
      What if we'll have 2TB of data on 2.5TB disk and we wanted to add
      another node and move tokens?<br>
      Is it possible to automate node addition to cluster and be sure we
      won't run out of space?<br>
      <br>
      Thank.<br>
    </blockquote>
  </body>
</html>

--------------030906040704060604070708--