From user-return-36318-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Tue Sep 3 16:48:16 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 20AF2106F5 for ; Tue, 3 Sep 2013 16:48:16 +0000 (UTC) Received: (qmail 98510 invoked by uid 500); 3 Sep 2013 16:48:13 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 98298 invoked by uid 500); 3 Sep 2013 16:48:13 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 98290 invoked by uid 99); 3 Sep 2013 16:48:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Sep 2013 16:48:12 +0000 X-ASF-Spam-Status: No, hits=3.3 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,TRACKER_ID,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [216.82.243.202] (HELO mail1.bemta8.messagelabs.com) (216.82.243.202) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Sep 2013 16:48:07 +0000 Received: from [216.82.241.196:30625] by server-10.bemta-8.messagelabs.com id 0C/3E-20977-D9216225; Tue, 03 Sep 2013 16:47:25 +0000 X-Env-Sender: Chris.Lohfink@digi.com X-Msg-Ref: server-3.tower-46.messagelabs.com!1378226844!30289559!1 X-Originating-IP: [66.77.174.13] X-StarScan-Received: X-StarScan-Version: 6.9.11; banners=-,-,- X-VirusChecked: Checked Received: (qmail 2440 invoked from network); 3 Sep 2013 16:47:25 -0000 Received: from mail.mx3.digi.com (HELO mcl-sms-ns1.digi.com) (66.77.174.13) by server-3.tower-46.messagelabs.com with RC4-SHA encrypted SMTP; 3 Sep 2013 16:47:25 -0000 Received: from mcl-sms-exch01.digi.com (10.5.8.50) by mail.mx1.digi.com (172.16.1.13) with Microsoft SMTP Server (TLS) id 8.3.327.1; Tue, 3 Sep 2013 11:47:20 -0500 Received: from mtk-sms-exch01.digi.com ([10.10.8.100]) by mcl-sms-exch01.digi.com ([10.5.8.50]) with mapi; Tue, 3 Sep 2013 11:47:24 -0500 From: "Lohfink, Chris" To: "user@cassandra.apache.org" Date: Tue, 3 Sep 2013 11:47:30 -0500 Subject: RE: read ? Thread-Topic: read ? Thread-Index: AQHOqKhF/UGE2JTS20azxDGk9lbrxZm0LhHQ///D/YCAAEWeoA== Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_D7797C4132D4A7458DA14B70BB00708AE173E42258mtksmsexch01d_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_D7797C4132D4A7458DA14B70BB00708AE173E42258mtksmsexch01d_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Does it actually OOM eventually? There will be a certain amount of object a= llocation for reads (or anything) which will see the heap creep up until a = GC, but at ~500mb or so of a 8gb heap there is little reason for the JVM to= do it so it probably just ignores it to save processing. Even the young g= en wont require a collection at this size. Which version of Cassandra are you running? Previous to 1.2 a lot of metada= ta about the sstables took considerable heap which could cause additional m= emory utilization. Chris From: Langston, Jim [mailto:Jim.Langston@compuware.com] Sent: Tuesday, September 03, 2013 11:33 AM To: user@cassandra.apache.org Subject: Re: read ? Thanks Chris, I have about 8 heap dumps that I have been looking at. I have been trying t= o isolate as to why I have be dumping heap, I've started by removing the apps that wr= ite to cassandra and eliminating work that would entail. I am left with just the a= pps that are reading the data and from the heap dumps it looks like Cassandra Column= methods being called, because there are so many objects, it is difficult to ascerta= in exactly what the problem may be. That prompted my query, trying to quickly determine if = Cassandra holds objects that have been used for reading, and if so, why, and more imp= ortantly if something can be done. Jim From: "Lohfink, Chris" > Reply-To: > Date: Tue, 3 Sep 2013 11:12:19 -0500 To: "user@cassandra.apache.org" > Subject: RE: read ? To get an accurate picture you should force a full GC on each node, the hea= p utilization can be misleading since there can be a lot of things in the h= eap with no strong references. There is a number of factors that can lead to this. For a true comparison = I would recommend using jconsole and call dumpHeap on com.sun.management:ty= pe=3DHotSpotDiagnostic with the 2nd param true (force GC). Then open the h= eap dump up in a tool like yourkit and you will get a better comparison and= also it will tell you what it is that's taking the space. Chris From: Langston, Jim [mailto:Jim.Langston@compuware.com] Sent: Tuesday, September 03, 2013 8:20 AM To: user@cassandra.apache.org Subject: read ? Hi all, Quick question I currently am looking at a 4 node cluster and I have currently stopped all= writing to Cassandra, with the reads continuing. I'm trying to understand the utiliza= tion of memory within the JVM. nodetool info on each of the nodes shows them all growing in footprint, 2 of the three at a greater rate. On the restart of C= assandra each were at about 100MB, after 2 days, each of the following are at: Heap Memory (MB) : 798.41 / 3052.00 Heap Memory (MB) : 370.44 / 3052.00 Heap Memory (MB) : 549.73 / 3052.00 Heap Memory (MB) : 481.89 / 3052.00 Ring configuration: Address Rack Status State Load Owns = Token = 127605887595351923798765477786913079296 x 1d Up Normal 4.38 GB 25.00% 0 x 1d Up Normal 4.17 GB 25.00% 42535295= 865117307932921825928971026432 x 1d Up Normal 4.19 GB 25.00% 85070591= 730234615865843651857942052864 x 1d Up Normal 4.14 GB 25.00% 12760588= 7595351923798765477786913079296 What I'm not sure of is what the growth is different between each ? and why that growth is being created by activity that is read only. Is Cassandra caching and holding the read data ? I currently have caching turned off for the key/row. Also as part of the in= fo command Key Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, = NaN recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, = NaN recent hit rate, 0 save period in seconds Thanks, Jim --_000_D7797C4132D4A7458DA14B70BB00708AE173E42258mtksmsexch01d_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Does it a= ctually OOM eventually? There will be a certain amount of object allocation= for reads (or anything) which will see the heap creep up until a GC, but a= t ~500mb or so of a 8gb heap there is little reason for the JVM to do it so= it probably just ignores it to save processing.  Even the young gen w= ont require a collection at this size.

 

Whi= ch version of Cassandra are you running? Previous to 1.2 a lot of metadata = about the sstables took considerable heap which could cause additional memo= ry utilization.

 = ;

Chris

 

From: Langston, Jim [mailto:Jim.Langston@compuware.com]
= Sent: Tuesday, September 03, 2013 11:33 AM
To: user@cassandra= .apache.org
Subject: Re: read ?

=

 

Th= anks Chris,

=  

I have about 8 = heap dumps that I have been looking at. I have been trying to isolate<= /o:p>

as to why I have be du= mping heap, I've started by removing the apps that write to

cassandra and eliminating work t= hat would entail. I am left with just the apps that =

are reading the data and from the = heap dumps it looks like Cassandra Column methods

being called, because there are so many ob= jects, it is difficult to ascertain exactly what

the problem may be. That prompted my query,= trying to quickly determine if Cassandra

<= p class=3DMsoNormal>holds objects that have been used for reading, and= if so, why, and more importantly if

something can be done.

=

 

Jim

=  

<= p class=3DMsoNormal> 

To get an accurate picture you should force a full GC= on each node, the heap utilization can be misleading since there can be a = lot of things in the heap with no strong references.

 =

There is a number of factors that can lead to this.  For a true com= parison I would recommend using jconsole and call dumpHeap on com.sun.manag= ement:type=3DHotSpotDiagnostic with the 2nd param true (force GC= ).  Then open the heap dump up in a tool like yourkit and you will get= a better comparison and also it will tell you what it is that’s taki= ng the space.

 =

Chris

 

From: Langston, Jim [mailto:Jim.Langston@compuware.com]
Sent: Tue= sday, September 03, 2013 8:20 AM
To: user@cassandra.apache.org
Subject: read ?

 

<= p class=3DMsoNormal>Hi all,

 

Quick question

 =

I currently am l= ooking at a 4 node cluster and I have currently stopped all writing to = ;

Cassandra,  with the reads continuing. I'm trying = to understand the utilization=

of memory within the JVM. n= odetool info on each of the nodes shows them all 

gr= owing in footprint, 2 of the three at a greater rate. On the restart of Cas= sandra

<= p class=3DMsoNormal>each were at about 100MB, after 2 days, each of th= e following are at:

 

Heap Memory = (MB) : 798.41 / 3052.00

 

Heap M= emory (MB) : 370.44 / 3052.00=

 

He= ap Memory (MB) : 549.73 / 3052.00

 

Heap Memory (MB) : 481.89 / 3052.00

 

Ring configuration:

 

A= ddress         Rack        Status S= tate   Load            Owns    = ;            Token

 =                     &nbs= p;                     &n= bsp;                     =              127605887595351923798765477= 786913079296

=

x     1d       &nbs= p;  Up     Normal  4.38 GB         = 25.00%              0

x   1d          Up     Normal &nbs= p;4.17 GB         25.00%         &n= bsp;    42535295865117307932921825928971026432

x   1d        = ;  Up     Normal  4.19 GB         2= 5.00%              850705917302346158658= 43651857942052864

<= /div>

x   <= /span>1d          Up     Normal  4.= 14 GB         25.00%          =    127605887595351923798765477786913079296

 

 <= /o:p>

What I'm not sure of i= s what the growth is different between each ? and why

tha= t growth is being created by activity that is read only.

 

=

Is Cassandra caching and holding the read data ?&= nbsp;

 

I currently have caching t= urned off for the key/row. Also as part of the info command

 

Key Cache        : siz= e 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 1= 4400 save period in seconds

Row Cache      = ;  : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recen= t hit rate, 0 save period in seconds

 <= span style=3D'color:black'>

 

<= /div>

 

Thanks,=

 

=

Jim=

= --_000_D7797C4132D4A7458DA14B70BB00708AE173E42258mtksmsexch01d_--