incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Haddad <...@jonhaddad.com>
Subject Re: cassandra performance problems
Date Thu, 05 Dec 2013 16:42:50 GMT
Do you mean high CPU usage or high load avg?  (20 indicates load avg to
me).  High load avg means the CPU is waiting on something.

Check "iostat -dmx 1 100" to check your disk stats, you'll see the columns
that indicate mb/s read & write as well as % utilization.

Once you understand the bottleneck we can start to narrow down the cause.


On Thu, Dec 5, 2013 at 4:33 AM, Alexander Shutyaev <shutyaev@gmail.com>wrote:

> Hi all,
>
> We have a 3 node cluster setup, single keyspace, about 500 tables. The
> hardware is 2 cores + 16 GB RAM (Cassandra chose to have 4GB). Cassandra
> version is 2.0.3. Our replication factor is 3, read/write consistency is
> QUORUM. We've plugged it into our production environment as a cache in
> front of postgres. Everything worked fine, we even stressed it by
> explicitly propagating about 30G (10G/node) data from postgres to cassandra.
>
> Then the problems came. Our nodes began showing high cpu usage (around
> 20). The funny thing is that they were actually doing it one after another
> and there was always only node with high cpu usage. Using OpsCenter we saw
> that when the CPU was beginning to go high the node in question was
> performing compaction. But even after the compaction was performed the cpu
> remained still high, and in some cases didn't go down for hours. Our jmx
> monitoring showed that it was presumably in constant garbage collection.
> During that time cluster read latency goes from 2ms to 200ms
>
> What can be the reason? Can it be high number of tables? Do we need to
> adjust some settings for this setup? Is it ok to have so many tables?
> Theoretically we can stuck them all in 3-4 tables.
>
> Thanks in advance,
> Alexander
>



-- 
Jon Haddad
http://www.rustyrazorblade.com
skype: rustyrazorblade

Mime
View raw message