Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A54271064E for ; Tue, 3 Sep 2013 16:34:20 +0000 (UTC) Received: (qmail 62016 invoked by uid 500); 3 Sep 2013 16:34:17 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 61972 invoked by uid 500); 3 Sep 2013 16:34:17 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 61934 invoked by uid 99); 3 Sep 2013 16:34:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Sep 2013 16:34:15 +0000 X-ASF-Spam-Status: No, hits=3.3 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,TRACKER_ID X-Spam-Check-By: apache.org Received-SPF: error (nike.apache.org: encountered temporary error during SPF processing of domain of Jim.Langston@compuware.com) Received: from [216.32.181.183] (HELO ch1outboundpool.messaging.microsoft.com) (216.32.181.183) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Sep 2013 16:34:05 +0000 Received: from mail115-ch1-R.bigfish.com (10.43.68.231) by CH1EHSOBE022.bigfish.com (10.43.70.79) with Microsoft SMTP Server id 14.1.225.22; Tue, 3 Sep 2013 16:33:23 +0000 Received: from mail115-ch1 (localhost [127.0.0.1]) by mail115-ch1-R.bigfish.com (Postfix) with ESMTP id C817E2000C9 for ; Tue, 3 Sep 2013 16:33:23 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.101;KIP:(null);UIP:(null);IPV:NLI;H:BL2PRD0510HT003.namprd05.prod.outlook.com;RD:none;EFVD:NLI X-SpamScore: 1 X-BigFish: PS1(zz9371Ic85eh4015I14ffI9a6kzz1f42h208ch1ee6h1de0h1fdah2073h1202h1e76h1d1ah1d2ah1fc6hzz1d7338h1de098h17326ah18c673h1de097h186068h8275bh8275dhz2ei2a8h839hbe3he5bhf0ah1288h12a5h12bdh137ah1441h1504h1537h153bh162dh1631h1758h18e1h1946h19b5h1ad9h1b0ah1bceh1d0ch1d2eh1d3fh1dc1h1dfeh1dffh1e1dh1fe8h1ff5h209eh1155h) Received-SPF: neutral (mail115-ch1: 157.56.240.101 is neither permitted nor denied by domain of compuware.com) client-ip=157.56.240.101; envelope-from=Jim.Langston@compuware.com; helo=BL2PRD0510HT003.namprd05.prod.outlook.com ;.outlook.com ; X-Forefront-Antispam-Report-Untrusted: SFV:NSPM;SFS:(377454003)(53754006)(164054003)(199002)(189002)(50986001)(47976001)(51856001)(46102001)(74366001)(74706001)(74876001)(15202345003)(16236675002)(47736001)(19300405004)(49866001)(4396001)(53806001)(54356001)(83506001)(80022001)(81816001)(79102001)(77096001)(63696002)(81686001)(47446002)(15975445006)(65816001)(74502001)(69226001)(56776001)(76176001)(83072001)(54316002)(59766001)(77982001)(76482001)(81342001)(56816003)(19580395003)(31966008)(36756003)(74662001)(19580405001)(83322001)(76796001)(76786001)(81542001)(80976001)(24704002);DIR:OUT;SFP:;SCL:1;SRVR:BN1PR05MB424;H:BN1PR05MB424.namprd05.prod.outlook.com;CLIP:10.255.100.4;RD:InfoNoRecords;A:1;MX:1;LANG:en; Received: from mail115-ch1 (localhost.localdomain [127.0.0.1]) by mail115-ch1 (MessageSwitch) id 1378226001163244_28337; Tue, 3 Sep 2013 16:33:21 +0000 (UTC) Received: from CH1EHSMHS009.bigfish.com (snatpool2.int.messaging.microsoft.com [10.43.68.231]) by mail115-ch1.bigfish.com (Postfix) with ESMTP id 22B971E0041 for ; Tue, 3 Sep 2013 16:33:21 +0000 (UTC) Received: from BL2PRD0510HT003.namprd05.prod.outlook.com (157.56.240.101) by CH1EHSMHS009.bigfish.com (10.43.70.9) with Microsoft SMTP Server (TLS) id 14.16.227.3; Tue, 3 Sep 2013 16:33:19 +0000 Received: from BN1PR05MB424.namprd05.prod.outlook.com (10.141.58.148) by BL2PRD0510HT003.namprd05.prod.outlook.com (10.255.100.38) with Microsoft SMTP Server (TLS) id 14.16.353.4; Tue, 3 Sep 2013 16:33:17 +0000 Received: from BN1PR05MB424.namprd05.prod.outlook.com (10.141.58.148) by BN1PR05MB424.namprd05.prod.outlook.com (10.141.58.148) with Microsoft SMTP Server (TLS) id 15.0.745.25; Tue, 3 Sep 2013 16:33:15 +0000 Received: from BN1PR05MB424.namprd05.prod.outlook.com ([169.254.8.106]) by BN1PR05MB424.namprd05.prod.outlook.com ([169.254.8.106]) with mapi id 15.00.0745.000; Tue, 3 Sep 2013 16:33:15 +0000 From: "Langston, Jim" To: "user@cassandra.apache.org" Subject: Re: read ? Thread-Topic: read ? Thread-Index: AQHOqKhF/UGE2JTS20azxDGk9lbrxZm0LhHQ///D/YA= Date: Tue, 3 Sep 2013 16:33:15 +0000 Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.14.0.111121 x-originating-ip: [10.255.100.4] x-forefront-prvs: 09583628E0 Content-Type: multipart/alternative; boundary="_000_CE4B865B398B9jimlangstoncompuwarecom_" MIME-Version: 1.0 X-MS-Exchange-CrossPremises-AuthAs: Internal X-MS-Exchange-CrossPremises-AuthMechanism: 04 X-MS-Exchange-CrossPremises-AuthSource: BN1PR05MB424.namprd05.prod.outlook.com X-MS-Exchange-CrossPremises-SCL: 1 X-MS-Exchange-CrossPremises-messagesource: StoreDriver X-MS-Exchange-CrossPremises-BCC: X-MS-Exchange-CrossPremises-originalclientipaddress: 10.255.100.4 X-MS-Exchange-CrossPremises-avstamp-service: 1.0 X-MS-Exchange-CrossPremises-antispam-scancontext: DIR:Originating;SFV:NSPM;SKIP:0; X-MS-Exchange-CrossPremises-processed-by-journaling: Journal Agent X-MS-Exchange-CrossPremises-ContentConversionOptions: False;00160000;True;;iso-8859-1 X-OrganizationHeadersPreserved: BN1PR05MB424.namprd05.prod.outlook.com X-OriginatorOrg: compuware.com X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% X-Virus-Checked: Checked by ClamAV on apache.org --_000_CE4B865B398B9jimlangstoncompuwarecom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Thanks Chris, I have about 8 heap dumps that I have been looking at. I have been trying t= o isolate as to why I have be dumping heap, I've started by removing the apps that wr= ite to cassandra and eliminating work that would entail. I am left with just the a= pps that are reading the data and from the heap dumps it looks like Cassandra Column= methods being called, because there are so many objects, it is difficult to ascerta= in exactly what the problem may be. That prompted my query, trying to quickly determine if = Cassandra holds objects that have been used for reading, and if so, why, and more imp= ortantly if something can be done. Jim From: "Lohfink, Chris" > Reply-To: > Date: Tue, 3 Sep 2013 11:12:19 -0500 To: "user@cassandra.apache.org" > Subject: RE: read ? To get an accurate picture you should force a full GC on each node, the hea= p utilization can be misleading since there can be a lot of things in the h= eap with no strong references. There is a number of factors that can lead to this. For a true comparison = I would recommend using jconsole and call dumpHeap on com.sun.management:ty= pe=3DHotSpotDiagnostic with the 2nd param true (force GC). Then open the h= eap dump up in a tool like yourkit and you will get a better comparison and= also it will tell you what it is that=92s taking the space. Chris From: Langston, Jim [mailto:Jim.Langston@compuware.com] Sent: Tuesday, September 03, 2013 8:20 AM To: user@cassandra.apache.org Subject: read ? Hi all, Quick question I currently am looking at a 4 node cluster and I have currently stopped all= writing to Cassandra, with the reads continuing. I'm trying to understand the utiliza= tion of memory within the JVM. nodetool info on each of the nodes shows them all growing in footprint, 2 of the three at a greater rate. On the restart of C= assandra each were at about 100MB, after 2 days, each of the following are at: Heap Memory (MB) : 798.41 / 3052.00 Heap Memory (MB) : 370.44 / 3052.00 Heap Memory (MB) : 549.73 / 3052.00 Heap Memory (MB) : 481.89 / 3052.00 Ring configuration: Address Rack Status State Load Owns = Token = 127605887595351923798765477786913079296 x 1d Up Normal 4.38 GB 25.00% 0 x 1d Up Normal 4.17 GB 25.00% 42535295= 865117307932921825928971026432 x 1d Up Normal 4.19 GB 25.00% 85070591= 730234615865843651857942052864 x 1d Up Normal 4.14 GB 25.00% 12760588= 7595351923798765477786913079296 What I'm not sure of is what the growth is different between each ? and why that growth is being created by activity that is read only. Is Cassandra caching and holding the read data ? I currently have caching turned off for the key/row. Also as part of the in= fo command Key Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, = NaN recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, = NaN recent hit rate, 0 save period in seconds Thanks, Jim --_000_CE4B865B398B9jimlangstoncompuwarecom_ Content-Type: text/html; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable
Thanks Chris,

I have about 8 heap dumps that I have been looking at. I have been try= ing to isolate
as to why I have be dumping heap, I've started by removing the apps th= at write to
cassandra and eliminating work that would entail. I am left with just = the apps that 
are reading the data and from the heap dumps it looks like Cassandra C= olumn methods
being called, because there are so many objects, it is difficult to as= certain exactly what
the problem may be. That prompted my query, trying to quickly determin= e if Cassandra
holds objects that have been used for reading, and if so, why, and mor= e importantly if
something can be done.

Jim

From: "Lohfink, Chris" &l= t;Chris.Lohfink@digi.com><= br> Reply-To: <user@cassandra.apache.org>
Date: Tue, 3 Sep 2013 11:12:19 -050= 0
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: RE: read ?

To get an accurate picture you sho= uld force a full GC on each node, the heap utilization can be misleading si= nce there can be a lot of things in the heap with no strong references.

 

There is a number of factors that = can lead to this.  For a true comparison I would recommend using jcons= ole and call dumpHeap on com.sun.management:type=3DHotSpotDiagnostic with the 2nd param true (force GC).  Then open the heap du= mp up in a tool like yourkit and you will get a better comparison and also = it will tell you what it is that=92s taking the space.

 

Chris

 

From: Langston, Jim [mailto:Jim.Langston@compuware.com]
Sent: Tuesday, September 03, 2013 8:20 AM
To: user@cassandra.apac= he.org
Subject: read ?

 

Hi all,

 

Quick question

 

I currently am looking at a 4 node cluster = and I have currently stopped all writing to 

Cassandra,  with the reads continuing.= I'm trying to understand the utilization

of memory within the JVM. nodetool info on = each of the nodes shows them all 

growing in footprint, 2 of the three at a g= reater rate. On the restart of Cassandra

each were at about 100MB, after 2 days, eac= h of the following are at:

 

Heap Memory (MB) : 798.41 / 3052.00

 

Heap Memory (MB) : 370.44 / 3052.00

 

Heap Memory (MB) : 549.73 / 3052.00

 

Heap Memory (MB) : 481.89 / 3052.00

 

Ring configuration:

 

Address         Rack &n= bsp;      Status State   Load       &nbs= p;    Owns                = ;Token

            &= nbsp;                    =                     &nbs= p;                     &n= bsp;  127605887595351923798765477786913079296

x     1d       &nb= sp;  Up     Normal  4.38 GB        = 25.00%              0=

x   1d         &nb= sp;Up     Normal  4.17 GB         25.00%=              42535295865117307932921825= 928971026432

x   1d          Up     Normal  4= .19 GB         25.00%          = ;    85070591730234615865843651857942052864

x   1d          Up     Normal  4= .14 GB         25.00%          = ;    127605887595351923798765477786913079296

 

 

What I'm not sure of is what the growth is = different between each ? and why

that growth is being created by activity th= at is read only.

 

Is Cassandra caching and holding the read d= ata ? 

 

I currently have caching turned off for the= key/row. Also as part of the info command

 

Key Cache        : size= 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 14= 400 save period in seconds

Row Cache        : size= 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 = save period in seconds

 

 

 

Thanks,

 

Jim

 --_000_CE4B865B398B9jimlangstoncompuwarecom_--