From user-return-12769-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Wed Feb 02 18:39:10 2011 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 77540 invoked from network); 2 Feb 2011 18:39:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 2 Feb 2011 18:39:10 -0000 Received: (qmail 16041 invoked by uid 500); 2 Feb 2011 18:39:08 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 15899 invoked by uid 500); 2 Feb 2011 18:39:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 15891 invoked by uid 99); 2 Feb 2011 18:39:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Feb 2011 18:39:05 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ryan@twitter.com designates 209.85.214.44 as permitted sender) Received: from [209.85.214.44] (HELO mail-bw0-f44.google.com) (209.85.214.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Feb 2011 18:38:59 +0000 Received: by bwz12 with SMTP id 12so923239bwz.31 for ; Wed, 02 Feb 2011 10:38:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=twitter.com; s=google; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=Mo1w2Y5jqhjzrKknxAXWljCoFNU0odUdlOkllerwPxs=; b=io3yzxnCBQwpydPEG9w29DhMExnfLs7C73womONuWpYaYdnSg0oVVOqYlmyrSZXipl c789HNI25ph+xPYXAihAMBPGlpQkVFzDJ55iPBWsdDMX2jrPvBTKFRzfowjnfu+lxwoC tBlyBIstP1sQ9FHA3dmvjREEFr310IqPKPgws= DomainKey-Signature: a=rsa-sha1; c=nofws; d=twitter.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=OrqmlreynAiKhsACbmg4Pqv0i1nlI/ul5subwmFowOOjfBorq/UuuNd7e1PtJg5iXA /EK2RWrmVufvjAdJCHjN6jtldGbVAjx19+gj4MD8Nnw0hExfjpo97mHkJeZlACIo0Z8R 82HetAXjSVxOjx5cTj3/A793jx8i/gHRYIpN8= MIME-Version: 1.0 Received: by 10.204.103.205 with SMTP id l13mr8478287bko.160.1296671918339; Wed, 02 Feb 2011 10:38:38 -0800 (PST) Received: by 10.204.69.8 with HTTP; Wed, 2 Feb 2011 10:38:38 -0800 (PST) In-Reply-To: <4D49A289.8000504@gmail.com> References: <4D41A9FB.4070300@gmail.com> <4D437940.6070500@gmail.com> <4D496896.1050702@gmail.com> <4D49A289.8000504@gmail.com> Date: Wed, 2 Feb 2011 10:38:38 -0800 Message-ID: Subject: Re: reduced cached mem; resident set size growth From: Ryan King To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org On Wed, Feb 2, 2011 at 10:29 AM, Chris Burroughs wrote: > On 02/02/2011 12:49 PM, Ryan King wrote: >> We're seeing a similar problem with one of our clusters (but over a >> longer time scale). Its possible that its not a leak, but just >> fragmentation. Unless you've told it otherwise, the jvm uses glibc's >> malloc implementation for off-heap allocations. We're currently >> running a test with jemalloc on one node to see if the problem goes >> away. >> > > Thanks Ryan. > > Is it over a longer time scale because of some action taken to mitigate > the problem, or has it always been that long for you? My guess is that its a longer timeframe because the cluster is really low traffic (around 100qps across 10 nodes). -ryan