Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of tomerbd1@gmail.com designates
 209.85.216.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        b=S/lYWUVPWYS1m1ba+zesvpMfxosjfcdHuOzY4xNT7JbNX37fgG92abXIXeSlirEFBT
         LMMXbVW3JXb91hPVMZkibqZHwAY7wG4d2VDYGgjJ5GaGJvcA8HAsCGltH1gIfzcxzEg2
         sxaNRp41ZN60WZRYWu0YwwOlxp4V0T5yXdiyQ=
MIME-Version: 1.0
From: Tomer B <tomerbd1@gmail.com>
Date: Thu, 12 May 2011 10:49:22 +0300
Message-ID: <BANLkTim-skGj4PH1tjK7y0XJttM6mfgDWg@mail.gmail.com>
Subject: Knowing when there is a *real* need to add nodes
To: user@cassandra.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi

I'm trying to predict when my cluster would soon be needing new nodes
added, i want a continuous graph telling my of my cluster health so
that when i see my cluster becomes more and more busy (I want numbers
& measurments) i would be able to know i need to start purchasing more
machines and get them into my cluster, so i want to know of that
beforehand.
I'm writing here what I came with after doing some research over net.
I would highly appreciate any additional gauge measurements and ranges
in order to test my cluster health and to know beforehand when i'm
going to soon need more nodes.Although i'm writing down green
gauge,yellow gauge,red gauge, i'm also trying to find a continuous
graph where i can tell where our cluster stand (as much as
possible...)

Also my recommendation is always before adding new nodes:

1. Make sure all nodes are balanced and if not balance them.
2. Separate commit log drive from data (SSTables) drive
3. use mmap index only in memory and not auto
4. Increase disk IO if possible.
5. Avoid swapping as much as possible.


As for my gauge tests for when to add new nodes:

test: nodetool tpstats -h <cassandra_host>
green gauge: No pending column with number higher
yellow gauge: pending columns 100-2000
red gauge:Larger than 3000

test: iostat -x -n -p -z 5 10  and iostat -xcn 5
green gauge: kw/s + kr/s reaches is below 25% capacity of disk io
yellow gauge: 20%-50%
red gauge: 50%+

test: ostat -x -n -p -z 5 10 and check %b column
green gauge: less than 10%
yellow gauge:  10%-80%
red gauge: 90%+

test: nodetool cfstats --host localhost
green gauge: =93SSTable count=94 item does not continually grow over time
yellow gauge:
red gauge: =93SSTable count=94 item continually grows over time

test: ./nodetool cfstats --host localhost | grep -i pending
green gauge: 0-2
yellow gauge: 3-100
red gauge: 101+

I would highly appreciate any additional gauge measurements and ranges
in order to test my cluster health and to know ***beforehand*** when
i'm going to soon need more nodes.