cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: when should new nodes be added to a cluster
Date Mon, 02 Aug 2010 21:39:42 GMT
You may need to provide some more information on how many reads your sending to the cluster.
Also...

How many nodes do you have in the cluster ?
When you are seeing high response times on one node, what's the load like on the others
?
Is the data load evenly distributed around the cluster ?
Are your clients connecting to different nodes in the cluster ?

Perhaps that node is somehow out of sync with the others...
Anything odd happened in the cluster recently, such as one node going down ?
When was the last time you ran repair?

Aaron

On 03 Aug, 2010,at 06:47 AM, Artie Copeland <yeslinux.now@gmail.com> wrote:

> i have a question on what are the signs from cassandra that new nodes should be added
to the cluster.  We are currently seeing long read times from the one node that has about
70GB of data with 60GB in one column family.  we are using a replication factor of 3.  I have
tracked down the slow to occur when either row-read-stage or message-deserializer-pool is
high like atleast 4000.  my systems are 16core, 3 TB, 48GB mem servers.  we would like to
be able to use more of the server than just 70GB.
>
> The system is a realtime system that needs to scale quite large.  Our current heap size
is 25GB and are getting atleast 50% row cache hit rates.  Does it seem strange that cassandra
is not able to handle the work load?  We perform multislice gets when reading similar to twissandra
does.  this is to cut down on the network ops.  Looking at iostat it doesnt appear to have
alot of queued reads.
>
> What are others seeing when they have to add new nodes?  What data sizes are they seeing?
 This is needed so we can plan our growth and server purchase strategy.
>
> thanx
> Artie
>
> -- 
> http://yeslinux.org
> http://yestech.org

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
    • Unnamed multipart/related (inline, None, 0 bytes)
View raw message