Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of sdolgy@gmail.com designates
 209.85.220.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=sc0M6HT6WaBk446Ianp1ISecNxMrOru9D6eOcUMIa5FPM6x51sVHLVFmUH/VOcwHwW
         7URgSd1mLPKMwDsZMP0NWX9BT27snKnlbwHivN1NrE2tGkkUk+TSUI6mVFAK8WiYonQk
         IuWSjo01Qs2S1hzPqjdRPEutlo4zrai/c5JFE=
MIME-Version: 1.0
In-Reply-To: <BANLkTinXLRyy2AdN0q_6DS-dc==muGx8Ag@mail.gmail.com>
References: <BANLkTi=xcKBDaQ+Y+PChpqkRfGub-Co41w@mail.gmail.com>
 <BANLkTi=t7hOofAQ11k3bjC_kH1p_1243eg@mail.gmail.com>
 <BANLkTinXLRyy2AdN0q_6DS-dc==muGx8Ag@mail.gmail.com>
From: Sasha Dolgy <sdolgy@gmail.com>
Date: Wed, 1 Jun 2011 21:37:33 +0200
Message-ID: <BANLkTi=sSQ1nHtfHbcs2E0RnT9e-TFe9cQ@mail.gmail.com>
Subject: Re: cascading failures due to memory
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

and is there anything specific that could be causing the issue between
 Java SE 1.6.0_24 and 1.6.0_25 ?  All nodes are _24

up to 64% memory usage today

-sd

On Wed, Jun 1, 2011 at 9:30 PM, Sasha Dolgy <sdolgy@gmail.com> wrote:
> is there a specific string I should be looking for in the logs that
> isn't super obvious to me at the moment...
>
> On Tue, May 31, 2011 at 8:21 PM, Jonathan Ellis <jbellis@gmail.com> wrote=
:
>> The place to start is with the statistics Cassandra logs after each GC.
>>
>> On Tue, May 31, 2011 at 5:01 AM, Sasha Dolgy <sdolgy@gmail.com> wrote:
>>> hi everyone,
>>>
>>> the current nodes i have deployed (4) have all been working fine, with
>>> not a lot of data ... more reads than writes at the moment. =A0as i had
>>> monitoring disabled, when one node's OS killed the cassandra process
>>> due to out of memory problems ... that was fine. =A024 hours later,
>>> another node, 24 hours later, another node ...until finally, all 4
>>> nodes no longer had cassandra running.
>>>
>>> When all nodes are started fresh, CPU utilization is at about 21% on
>>> each box. =A0after 24 hours, this goes up to 32% and then 51% 24 hours
>>> later.
>>>
>>> originally I had thought that this may be a result of 'nodetool
>>> repair' not being run consistently ... after adding a cronjob to run
>>> every 24 hours (staggered between nodes) the problem of the increasing
>>> memory utilization does not resolve.
>>>
>>> i've read the operations page and also the
>>> http://wiki.apache.org/cassandra/MemtableThresholds page. =A0i am
>>> running defaults and 0.7.6-02 ...
>>>
>>> what are the best places to start in terms of finding why this is
>>> happening? =A0CF design / usage? =A0'nodetool cfstats' gives me some go=
od
>>> info ... and i've already implemented some changes to one CF based on
>>> how it had ballooned (too many rows versus not enough columns)
>>>
>>> suggestions appreciated
>>>
>>> --
>>> Sasha Dolgy
>>> sasha.dolgy@gmail.com
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
>
>
> --
> Sasha Dolgy
> sasha.dolgy@gmail.com
>


--=20
Sasha Dolgy
sasha.dolgy@gmail.com