Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9D8B1FDCD for ; Tue, 16 Apr 2013 21:42:01 +0000 (UTC) Received: (qmail 34100 invoked by uid 500); 16 Apr 2013 21:41:59 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 34065 invoked by uid 500); 16 Apr 2013 21:41:59 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 34057 invoked by uid 99); 16 Apr 2013 21:41:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2013 21:41:58 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a57.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2013 21:41:51 +0000 Received: from homiemail-a57.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a57.g.dreamhost.com (Postfix) with ESMTP id 41455208060 for ; Tue, 16 Apr 2013 14:41:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=vIilGoLHHQVi/5Cjdwwu5jYHZ5 k=; b=WwFkayCm7n9xOwcFcs9LQpsQTxTiqcAbjp4NLQ/nqgyJZRtSXBaHj0RPJB aiRDS06f0NmTBbHJKh1i8Yp5fdeLBQSHCi9aWixOQRAHOIdXJNRIijrTnUA89E6a 1HkTuraokRx6KmiIJVoAmJbQcp0kAC5VCNZoA53IKtU91uCIM= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a57.g.dreamhost.com (Postfix) with ESMTPSA id AE78C20801D for ; Tue, 16 Apr 2013 14:41:24 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_F488AB35-851E-4AA1-95BA-A966365375F3" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: C* consumes all RAM Date: Wed, 17 Apr 2013 09:41:28 +1200 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_F488AB35-851E-4AA1-95BA-A966365375F3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii You are probably seeing this http://wiki.apache.org/cassandra/FAQ#mmap Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 16/04/2013, at 8:43 PM, Mikhail Mazursky wrote: > More details: >=20 > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME = COMMAND=20 > 219 3801 0.7 92.7 6561116 3567016 ? SLl Mar11 372:44 = /usr/java/latest/bin/java >=20 > Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC = 2012 x86_64 x86_64 x86_64 GNU/Linux >=20 > total used free shared buffers cached=20 > Mem: 3754 3695 59 0 37 = 75=20 > -/+ buffers/cache: 3582 171=20 > Swap: 2055 165 1890 >=20 >=20 >=20 > 2013/4/16 Mikhail Mazursky > Hello. >=20 > C* have been running without any problem for some weeks but now it = started to consume all available ram. >=20 > The cluster have very little data in it. There are no errors in logs, = CPU is not loaded at all, jstack shows no deadlocks, there are 83 = threads. Read/write latency is 1-4ms. >=20 > The question is: how to find out where is the RAM used? >=20 > # nodetool ring keyspace name > Address DC Rack Status State Load = Effective-Ownership Token =20 > = 114967934986393518082067045023985443411 =20 > 10.0.1.220 datacenter1 rack1 Up Normal 889.71 KB = 100.00% 0 =20 > 10.0.1.222 datacenter1 rack1 Up Normal 918.92 KB = 100.00% 56662868904138943096229641828625627278 =20 > 10.0.1.221 datacenter1 rack1 Up Normal 1.04 MB = 100.00% 114967934986393518082067045023985443411 >=20 > We use AWS m1.medium - 3.75 RAM, 1 Core=20 >=20 > java version "1.7.0_10"=20 > Java(TM) SE Runtime Environment (build 1.7.0_10-b18)=20 > Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode) >=20 > C* is started with the following command > /usr/java/latest/bin/java -ea = -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar = -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=3D42 -Xms1G -Xmx1G = -Xmn512M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC >=20 > Thanks in advance. >=20 > Mikhail. >=20 --Apple-Mail=_F488AB35-851E-4AA1-95BA-A966365375F3 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii You = are probably seeing this http://wiki.apache.org/= cassandra/FAQ#mmap

Cheers

http://www.thelastpickle.com

On 16/04/2013, at 8:43 PM, Mikhail Mazursky <ash2kk@gmail.com> = wrote:

More = details:

USER       PID %CPU = %MEM    VSZ   RSS = TTY      STAT START   TIME COMMAND
219       3801  0.7 92.7 6561116 = 3567016 ?     SLl  Mar11 372:44 = /usr/java/latest/bin/java

Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 = #1 SMP Wed Oct 3 19:54:33 UTC 2012 x86_64 x86_64 x86_64 = GNU/Linux

total       = used       free     = shared    buffers     cached
Mem:          = 3754       = 3695         = 59          = 0         = 37         75
-/+ buffers/cache:       = 3582        171
Swap:         = 2055        = 165       1890



2013/4/16 = Mikhail Mazursky <ash2kk@gmail.com>
Hello.

C* have been = running without any problem for some weeks but now it started to consume = all available ram.

The cluster have very little data in it. There are no errors = in logs, CPU is not loaded at all, jstack shows no deadlocks, there are = 83 threads. Read/write latency is 1-4ms.

The question is: how to find out where is the RAM = used?

# nodetool ring keyspace name
Address         = DC          = Rack        Status State   = Load            = Effective-Ownership = Token           &nb= sp;            = ;            &= nbsp;   =
           &nbs= p;            =             &n= bsp;           &nbs= p;            =             &n= bsp;           &nbs= p;     = 114967934986393518082067045023985443411     
10.0.1.220      datacenter1 = rack1       Up     = Normal  889.71 KB       = 100.00%           &= nbsp; = 0            &= nbsp;           &nb= sp;            = ;      
10.0.1.222      datacenter1 = rack1       Up     = Normal  918.92 KB       = 100.00%           &= nbsp; = 56662868904138943096229641828625627278      =
10.0.1.221      datacenter1 = rack1       Up     = Normal  1.04 MB         = 100.00%           &= nbsp; 114967934986393518082067045023985443411

We use AWS = m1.medium - 3.75 RAM, 1 Core

java version "1.7.0_10"
Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed = mode)

C* is started with the following = command
/usr/java/latest/bin/java -ea = -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar = -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=3D42 -Xms1G -Xmx1G = -Xmn512M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC

Thanks in advance.

Mikhail.


= --Apple-Mail=_F488AB35-851E-4AA1-95BA-A966365375F3--