incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Watanabe Maki <watanabe.m...@gmail.com>
Subject Re: Knowing when there is a *real* need to add nodes
Date Thu, 12 May 2011 09:45:21 GMT
It's interesting topic for me too.
How about to add measurement on static disk utilization (% used) and memory utilization (
rss, JVM heap, JVM GC )?

maki

From iPhone


On 2011/05/12, at 0:49, Tomer B <tomerbd1@gmail.com> wrote:

> Hi
> 
> I'm trying to predict when my cluster would soon be needing new nodes
> added, i want a continuous graph telling my of my cluster health so
> that when i see my cluster becomes more and more busy (I want numbers
> & measurments) i would be able to know i need to start purchasing more
> machines and get them into my cluster, so i want to know of that
> beforehand.
> I'm writing here what I came with after doing some research over net.
> I would highly appreciate any additional gauge measurements and ranges
> in order to test my cluster health and to know beforehand when i'm
> going to soon need more nodes.Although i'm writing down green
> gauge,yellow gauge,red gauge, i'm also trying to find a continuous
> graph where i can tell where our cluster stand (as much as
> possible...)
> 
> Also my recommendation is always before adding new nodes:
> 
> 1. Make sure all nodes are balanced and if not balance them.
> 2. Separate commit log drive from data (SSTables) drive
> 3. use mmap index only in memory and not auto
> 4. Increase disk IO if possible.
> 5. Avoid swapping as much as possible.
> 
> 
> As for my gauge tests for when to add new nodes:
> 
> test: nodetool tpstats -h <cassandra_host>
> green gauge: No pending column with number higher
> yellow gauge: pending columns 100-2000
> red gauge:Larger than 3000
> 
> test: iostat -x -n -p -z 5 10  and iostat -xcn 5
> green gauge: kw/s + kr/s reaches is below 25% capacity of disk io
> yellow gauge: 20%-50%
> red gauge: 50%+
> 
> test: ostat -x -n -p -z 5 10 and check %b column
> green gauge: less than 10%
> yellow gauge:  10%-80%
> red gauge: 90%+
> 
> test: nodetool cfstats --host localhost
> green gauge: “SSTable count” item does not continually grow over time
> yellow gauge:
> red gauge: “SSTable count” item continually grows over time
> 
> test: ./nodetool cfstats --host localhost | grep -i pending
> green gauge: 0-2
> yellow gauge: 3-100
> red gauge: 101+
> 
> I would highly appreciate any additional gauge measurements and ranges
> in order to test my cluster health and to know ***beforehand*** when
> i'm going to soon need more nodes.

Mime
View raw message