Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of ash2kk@gmail.com designates
 209.85.215.51 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGg1-iM+1cGFhOV+JxoW3dx5LQ_Y-qmXvNa0h7svjHzvTRW1LQ@mail.gmail.com>
References: 
 <CAGg1-iM+1cGFhOV+JxoW3dx5LQ_Y-qmXvNa0h7svjHzvTRW1LQ@mail.gmail.com>
From: Mikhail Mazursky <ash2kk@gmail.com>
Date: Tue, 16 Apr 2013 14:43:28 +0600
Message-ID: 
 <CAGg1-iM3A7J5rW8PN7N4kOmDXeVgSac7ZxgB3EEZKTK-Xq_Csw@mail.gmail.com>
Subject: Re: C* consumes all RAM
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a11c26b00372ca604da765eab

--001a11c26b00372ca604da765eab
Content-Type: text/plain; charset=UTF-8

More details:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
219       3801  0.7 92.7 6561116 3567016 ?     SLl  Mar11 372:44
/usr/java/latest/bin/java

Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux

total       used       free     shared    buffers     cached
Mem:          3754       3695         59          0         37         75
-/+ buffers/cache:       3582        171
Swap:         2055        165       1890


2013/4/16 Mikhail Mazursky <ash2kk@gmail.com>

> Hello.
>
> C* have been running without any problem for some weeks but now it started
> to consume all available ram.
>
> The cluster have very little data in it. There are no errors in logs, CPU
> is not loaded at all, jstack shows no deadlocks, there are 83 threads.
> Read/write latency is 1-4ms.
>
> The question is: how to find out where is the RAM used?
>
> # nodetool ring keyspace name
> Address         DC          Rack        Status State   Load
> Effective-Ownership Token
>
> 114967934986393518082067045023985443411
> 10.0.1.220      datacenter1 rack1       Up     Normal  889.71 KB
> 100.00%             0
> 10.0.1.222      datacenter1 rack1       Up     Normal  918.92 KB
> 100.00%             56662868904138943096229641828625627278
> 10.0.1.221      datacenter1 rack1       Up     Normal  1.04 MB
> 100.00%             114967934986393518082067045023985443411
>
> We use AWS m1.medium - 3.75 RAM, 1 Core
>
> java version "1.7.0_10"
> Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
> Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
>
> C* is started with the following command
> /usr/java/latest/bin/java -ea
> -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M
> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC
>
> Thanks in advance.
>
> Mikhail.
>

--001a11c26b00372ca604da765eab
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: base64

PGRpdiBkaXI9Imx0ciI+TW9yZSBkZXRhaWxzOjxicj48YnI+VVNFUsKgwqDCoMKgwqDCoCBQSUQg
JUNQVSAlTUVNwqDCoMKgIFZTWsKgwqAgUlNTIFRUWcKgwqDCoMKgwqAgU1RBVCBTVEFSVMKgwqAg
VElNRSBDT01NQU5EDQo8YnI+MjE5wqDCoMKgwqDCoMKgIDM4MDHCoCAwLjcgOTIuNyA2NTYxMTE2
IDM1NjcwMTYgP8KgwqDCoMKgIFNMbMKgIE1hcjExIDM3Mjo0NCAvdXNyL2phdmEvbGF0ZXN0L2Jp
bi9qYXZhPGJyPjxicj5MaW51eCBYWFhAWVlZIDMuMi4zMC00OS41OS5hbXpuMS54ODZfNjQgIzEg
U01QIFdlZCBPY3QgMyAxOTo1NDozMyBVVEMgMjAxMiB4ODZfNjQgeDg2XzY0IHg4Nl82NCBHTlUv
TGludXg8YnI+PGJyPnRvdGFswqDCoMKgwqDCoMKgIHVzZWTCoMKgwqDCoMKgwqAgZnJlZcKgwqDC
oMKgIHNoYXJlZMKgwqDCoCBidWZmZXJzwqDCoMKgwqAgY2FjaGVkDQo8YnI+TWVtOsKgwqDCoMKg
wqDCoMKgwqDCoCAzNzU0wqDCoMKgwqDCoMKgIDM2OTXCoMKgwqDCoMKgwqDCoMKgIDU5wqDCoMKg
wqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgIDM3wqDCoMKgwqDCoMKgwqDCoCA3NQ0KPGJy
Pi0vKyBidWZmZXJzL2NhY2hlOsKgwqDCoMKgwqDCoCAzNTgywqDCoMKgwqDCoMKgwqAgMTcxDQo8
YnI+U3dhcDrCoMKgwqDCoMKgwqDCoMKgIDIwNTXCoMKgwqDCoMKgwqDCoCAxNjXCoMKgwqDCoMKg
wqAgMTg5MDxicj48YnI+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxicj48YnI+PGRpdiBjbGFz
cz0iZ21haWxfcXVvdGUiPjIwMTMvNC8xNiBNaWtoYWlsIE1henVyc2t5IDxzcGFuIGRpcj0ibHRy
Ij4mbHQ7PGEgaHJlZj0ibWFpbHRvOmFzaDJra0BnbWFpbC5jb20iIHRhcmdldD0iX2JsYW5rIj5h
c2gya2tAZ21haWwuY29tPC9hPiZndDs8L3NwYW4+PGJyPg0KDQo8YmxvY2txdW90ZSBjbGFzcz0i
Z21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2Nj
YyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj48ZGl2PjxkaXY+PGRpdj48
ZGl2PjxkaXY+SGVsbG8uPGJyPjxicj48L2Rpdj5DKiBoYXZlIGJlZW4gcnVubmluZyB3aXRob3V0
IGFueSBwcm9ibGVtIGZvciBzb21lIHdlZWtzIGJ1dCBub3cgaXQgc3RhcnRlZCB0byBjb25zdW1l
IGFsbCBhdmFpbGFibGUgcmFtLjxicj4NCg0KPGJyPjwvZGl2PlRoZSBjbHVzdGVyIGhhdmUgdmVy
eSBsaXR0bGUgZGF0YSBpbiBpdC4gVGhlcmUgYXJlIG5vIGVycm9ycyBpbiBsb2dzLCBDUFUgaXMg
bm90IGxvYWRlZCBhdCBhbGwsIGpzdGFjayBzaG93cyBubyBkZWFkbG9ja3MsIHRoZXJlIGFyZSA4
MyB0aHJlYWRzLiBSZWFkL3dyaXRlIGxhdGVuY3kgaXMgMS00bXMuPGJyPg0KPGJyPjwvZGl2Pjxk
aXY+VGhlIHF1ZXN0aW9uIGlzOiBob3cgdG8gZmluZCBvdXQgd2hlcmUgaXMgdGhlIFJBTSB1c2Vk
Pzxicj48L2Rpdj48ZGl2Pjxicj4jIG5vZGV0b29sIHJpbmcga2V5c3BhY2UNCm5hbWU8YnI+QWRk
cmVzc8KgwqDCoMKgwqDCoMKgwqAgREPCoMKgwqDCoMKgwqDCoMKgwqAgUmFja8KgwqDCoMKgwqDC
oMKgIFN0YXR1cyBTdGF0ZcKgwqAgTG9hZMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgRWZmZWN0aXZl
LU93bmVyc2hpcCBUb2tlbsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC
oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoA0KPGJyPsKgwqDCoMKgwqDCoMKg
wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC
oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg
wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAxMTQ5
Njc5MzQ5ODYzOTM1MTgwODIwNjcwNDUwMjM5ODU0NDM0MTHCoMKgwqDCoMKgDQo8YnI+MTAuMC4x
LjIyMMKgwqDCoMKgwqAgZGF0YWNlbnRlcjEgcmFjazHCoMKgwqDCoMKgwqAgVXDCoMKgwqDCoCBO
b3JtYWzCoCA4ODkuNzEgS0LCoMKgwqDCoMKgwqAgMTAwLjAwJcKgwqDCoMKgwqDCoMKgwqDCoMKg
wqDCoCAwwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC
oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqANCjxicj4xMC4wLjEuMjIywqDCoMKg
wqDCoCBkYXRhY2VudGVyMSByYWNrMcKgwqDCoMKgwqDCoCBVcMKgwqDCoMKgIE5vcm1hbMKgIDkx
OC45MiBLQsKgwqDCoMKgwqDCoCAxMDAuMDAlwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDU2NjYy
ODY4OTA0MTM4OTQzMDk2MjI5NjQxODI4NjI1NjI3Mjc4wqDCoMKgwqDCoMKgDQo8YnI+MTAuMC4x
LjIyMcKgwqDCoMKgwqAgZGF0YWNlbnRlcjEgcmFjazHCoMKgwqDCoMKgwqAgVXDCoMKgwqDCoCBO
b3JtYWzCoCAxLjA0IE1CwqDCoMKgwqDCoMKgwqDCoCAxMDAuMDAlwqDCoMKgwqDCoMKgwqDCoMKg
wqDCoMKgIDExNDk2NzkzNDk4NjM5MzUxODA4MjA2NzA0NTAyMzk4NTQ0MzQxMTxicj48YnI+V2Ug
dXNlIEFXUyBtMS5tZWRpdW0gLSAzLjc1IFJBTSwgMSBDb3JlDQo8YnI+PGJyPmphdmEgdmVyc2lv
biAmcXVvdDsxLjcuMF8xMCZxdW90Ow0KPGJyPkphdmEoVE0pIFNFIFJ1bnRpbWUgRW52aXJvbm1l
bnQgKGJ1aWxkIDEuNy4wXzEwLWIxOCkNCjxicj5KYXZhIEhvdFNwb3QoVE0pIDY0LUJpdCBTZXJ2
ZXIgVk0gKGJ1aWxkIDIzLjYtYjA0LCBtaXhlZCBtb2RlKTxicj48YnI+PC9kaXY+QyogaXMgc3Rh
cnRlZCB3aXRoIHRoZSBmb2xsb3dpbmcgY29tbWFuZDxicj4vdXNyL2phdmEvbGF0ZXN0L2Jpbi9q
YXZhIC1lYSAtamF2YWFnZW50Oi91c3Ivc2hhcmUvY2Fzc2FuZHJhL2xpYi9qYW1tLTAuMi41Lmph
ciAtWFg6K1VzZVRocmVhZFByaW9yaXRpZXMgLVhYOlRocmVhZFByaW9yaXR5UG9saWN5PTQyIC1Y
bXMxRyAtWG14MUcgLVhtbjUxMk0gLVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvciAtWHNz
MjU2ayAtWFg6K1VzZVBhck5ld0dDPGJyPg0KDQoNCjxicj48L2Rpdj5UaGFua3MgaW4gYWR2YW5j
ZS48c3BhbiBjbGFzcz0iSE9FblpiIj48Zm9udCBjb2xvcj0iIzg4ODg4OCI+PGJyPjxicj48L2Zv
bnQ+PC9zcGFuPjwvZGl2PjxzcGFuIGNsYXNzPSJIT0VuWmIiPjxmb250IGNvbG9yPSIjODg4ODg4
Ij5NaWtoYWlsLjxicj48L2ZvbnQ+PC9zcGFuPjwvZGl2Pg0KPC9ibG9ja3F1b3RlPjwvZGl2Pjxi
cj48L2Rpdj48L2Rpdj4NCg==
--001a11c26b00372ca604da765eab--