Return-Path: 
 <user-return-13189-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org>
Delivered-To: apmail-cassandra-user-archive@www.apache.org
Received: (qmail 49744 invoked from network); 10 Feb 2011 16:34:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 16:34:34 -0000
Received: (qmail 17354 invoked by uid 500); 10 Feb 2011 16:34:32 -0000
Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org
Received: (qmail 17333 invoked by uid 500); 10 Feb 2011 16:34:29 -0000
Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@cassandra.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@cassandra.apache.org>
List-Post: <mailto:user@cassandra.apache.org>
List-Id: <user.cassandra.apache.org>
Reply-To: user@cassandra.apache.org
Delivered-To: mailing list user@cassandra.apache.org
Received: (qmail 17325 invoked by uid 99); 10 Feb 2011 16:34:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 16:34:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.172] (HELO mail-yx0-f172.google.com)
 (209.85.213.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 16:34:24 +0000
Received: by yxt33 with SMTP id 33so743066yxt.31
        for <user@cassandra.apache.org>; Thu, 10 Feb 2011 08:34:03 -0800 (PST)
MIME-Version: 1.0
Received: by 10.236.110.5 with SMTP id t5mr2104270yhg.8.1297355642168; Thu, 10
 Feb 2011 08:34:02 -0800 (PST)
Received: by 10.236.108.44 with HTTP; Thu, 10 Feb 2011 08:34:02 -0800 (PST)
In-Reply-To: <AANLkTiknDJdA+=xy6zT4y_+LCzsoLqh0Jx_M2kqx=nhv@mail.gmail.com>
References: <AANLkTikgo-iFGZ7qmneNqLQZ3wUCeLcYS2bnH5_=geqT@mail.gmail.com>
	<AANLkTimm1YFLnHs5OwaRPKdPeQT0366gf7HdQZHOZWYs@mail.gmail.com>
	<AANLkTimG6brYwS-zhgedmQGGd2td7jtWWRfRN65Acv+n@mail.gmail.com>
	<AANLkTiknDJdA+=xy6zT4y_+LCzsoLqh0Jx_M2kqx=nhv@mail.gmail.com>
Date: Thu, 10 Feb 2011 11:34:02 -0500
Message-ID: <AANLkTi=cuUPSQCcn76m2UdY3T9EPULOKSiMwAy226pTs@mail.gmail.com>
Subject: Re: Out of control memory consumption
From: Huy Le <huyle@springpartners.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=00235445baca32e87b049bf02651

--00235445baca32e87b049bf02651
Content-Type: text/plain; charset=ISO-8859-1

We use Cassandra version 0.6.11. Our cache size is very small.  11 out of 12
servers have used heap size less than 500MB of 3GB allocated. Just one
server that had memory usage run out of control.

The issue is isolated.  It turn out that one CF has a row with compacted row
size of 50MB.  And this row happens to be very active row that get updated
every few seconds.   By deleting this row, heap memory usage is consistently
low across servers on the cluster.

It came to my surprise how expensive it is to have this row of 50MB
compacted size being updated every few seconds.  Does anyone have any
insight as to why the row mentioned is so expensive to have?

Thanks!

Huy

On Wed, Feb 9, 2011 at 2:34 PM, Robert Coli <rcoli@digg.com> wrote:

> On Wed, Feb 9, 2011 at 11:04 AM, Huy Le <huyle@springpartners.com> wrote:
> > Memory usage grows overtime.
>
> It is relatively typical for caches to exert memory pressure over time
> as they fill. What are your cache settings, for how many
> columnfamilies, and with what sized memtables? What version of
> Cassandra?
>
> =Rob
>



-- 
Huy Le
Spring Partners, Inc.
http://springpadit.com

--00235445baca32e87b049bf02651
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

We use Cassandra version 0.6.11. Our cache size is very small.=A0 11 out of=
 12 servers have used heap size less than 500MB of 3GB allocated. Just one =
server that had memory usage run out of control.<br><br>The issue is isolat=
ed.=A0 It turn out that one CF has a row with compacted row size of 50MB.=
=A0 And this row happens to be very active row that get updated every few s=
econds.=A0=A0 By deleting this row, heap memory usage is consistently low a=
cross servers on the cluster.<br>
<br>It came to my surprise how expensive it is to have this row of 50MB com=
pacted size being updated every few seconds.=A0 Does anyone have any insigh=
t as to why the row mentioned is so expensive to have? <br><br>Thanks!<br>
<br>Huy<br><br><div class=3D"gmail_quote">On Wed, Feb 9, 2011 at 2:34 PM, R=
obert Coli <span dir=3D"ltr">&lt;<a href=3D"mailto:rcoli@digg.com">rcoli@di=
gg.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"=
margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); paddi=
ng-left: 1ex;">
On Wed, Feb 9, 2011 at 11:04 AM, Huy Le &lt;<a href=3D"mailto:huyle@springp=
artners.com">huyle@springpartners.com</a>&gt; wrote:<br>
&gt; Memory usage grows overtime.<br>
<br>
It is relatively typical for caches to exert memory pressure over time<br>
as they fill. What are your cache settings, for how many<br>
columnfamilies, and with what sized memtables? What version of<br>
Cassandra?<br>
<br>
=3DRob<br>
</blockquote></div><br><br clear=3D"all"><br>-- <br>Huy Le <br>Spring Partn=
ers, Inc.<br><a href=3D"http://springpadit.com">http://springpadit.com</a> =
<br>

--00235445baca32e87b049bf02651--