Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of sdolgy@gmail.com designates
 209.85.212.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=FJMk3e+h0H+fWESGmURJJWBrcxHoXumykKAXBgYcSfGhK98S5xvkF0LsgdFk7Xk0g7
         engFjDeOd7F5gwFoD11egfHbVxaLveKEjQlaGLDUj3xM0rgNzC6Z2oEhK3Ng4uH4nn2N
         7iLPfRkywS6q0/sQ+pboUpfRToNxZI80Y8Ua8=
MIME-Version: 1.0
From: Sasha Dolgy <sdolgy@gmail.com>
Date: Tue, 31 May 2011 12:01:53 +0200
Message-ID: <BANLkTi=xcKBDaQ+Y+PChpqkRfGub-Co41w@mail.gmail.com>
Subject: cascading failures due to memory
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1

hi everyone,

the current nodes i have deployed (4) have all been working fine, with
not a lot of data ... more reads than writes at the moment.  as i had
monitoring disabled, when one node's OS killed the cassandra process
due to out of memory problems ... that was fine.  24 hours later,
another node, 24 hours later, another node ...until finally, all 4
nodes no longer had cassandra running.

When all nodes are started fresh, CPU utilization is at about 21% on
each box.  after 24 hours, this goes up to 32% and then 51% 24 hours
later.

originally I had thought that this may be a result of 'nodetool
repair' not being run consistently ... after adding a cronjob to run
every 24 hours (staggered between nodes) the problem of the increasing
memory utilization does not resolve.

i've read the operations page and also the
http://wiki.apache.org/cassandra/MemtableThresholds page.  i am
running defaults and 0.7.6-02 ...

what are the best places to start in terms of finding why this is
happening?  CF design / usage?  'nodetool cfstats' gives me some good
info ... and i've already implemented some changes to one CF based on
how it had ballooned (too many rows versus not enough columns)

suggestions appreciated

-- 
Sasha Dolgy
sasha.dolgy@gmail.com