cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Heckman <...@pagerduty.com>
Subject Re: Cassandra DC2 nodes down after increasing write requests on DC1 nodes
Date Sun, 16 Nov 2014 18:14:22 GMT
Hello Gabriel,

On Sun, Nov 16, 2014 at 7:25 AM, Gabriel Menegatti
<gabriel@s1mbi0se.com.br> wrote:
> I said that load was not a big deal, because ops center shows this loads as
> green, not as yellow or red at all.
>
> Also, our servers have many processors/threads, so I *think* this load is
> not problematic.

I've seen Cassandra clusters fall over with less load on the boxes.
So, not sure how trusting I am of Opscenter.

However, the impact is dependent on the system resources you have
available to you. How many CPU cores do these systems have, how much
total and free memory, are the underlying disks SSD or spinning
platters of rust?

> My assumption is that for some reason the DC2 10 nodes are not being able to
> handle the volume of requests from DC1, as it was 30 nodes. Even so, on my
> point of view the load of the DC2 nodes should go really high before
> Cassandra goes down, but its not doing so.

That would make sense if the nodes are under-provisioned for the work
you are trying to throw at them. The load averages and OOM in the heap
seems to indicate that may be a problem. However, without more details
it's hard to say.

> Regards,
> Gabriel

Cheers!
-Tim

Mime
View raw message