Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 372B0DEDF for ; Mon, 4 Mar 2013 18:20:56 +0000 (UTC) Received: (qmail 72336 invoked by uid 500); 4 Mar 2013 18:20:53 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 72316 invoked by uid 500); 4 Mar 2013 18:20:53 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 72307 invoked by uid 99); 4 Mar 2013 18:20:53 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Mar 2013 18:20:53 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of tompoges@gmail.com designates 74.125.82.45 as permitted sender) Received: from [74.125.82.45] (HELO mail-wg0-f45.google.com) (74.125.82.45) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Mar 2013 18:20:46 +0000 Received: by mail-wg0-f45.google.com with SMTP id dq12so4544378wgb.24 for ; Mon, 04 Mar 2013 10:20:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=PGDkV1g5KAtPkKGK1m1iEZGfvxu4DqNTRGVvUb09NJE=; b=CL0VwfPBgVo4pnd8KofMzMNrDmzmeDHQN2e0zK8ElNE3DWJrpsx5vurdyGUOB7GO8v eHY77UnJNJgYk0Rci1KWAMtl5TazFLFOBMpL6RcLVzu9PUrKmb1R9ShVw4eUjimCL1K9 oO8pk/h0Ml9fvepDC1oVp2r4eIr0VIAZCmatz3Qrq3UTevlK66Yro/19JApf6QFO4dux G96bcQ2SN3rOaYjKAX1oZVbyiQak4jz2Vy8g1diNmwOXpnR2XaLx1i+CYKMtZe8XU1YG jRbzlL45uToaW2+9uvL9k8xMVCQrbo7OwEGpo3GttdQd0qcfbcfzcAhrOvjtvgqS+huf 6hyA== MIME-Version: 1.0 X-Received: by 10.180.104.10 with SMTP id ga10mr13616121wib.23.1362421226430; Mon, 04 Mar 2013 10:20:26 -0800 (PST) Received: by 10.194.13.232 with HTTP; Mon, 4 Mar 2013 10:20:26 -0800 (PST) In-Reply-To: References: Date: Mon, 4 Mar 2013 18:20:26 +0000 Message-ID: Subject: Re: Poor read latency From: Tom Martin To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d04374a093c6c0404d71d6908 X-Virus-Checked: Checked by ClamAV on apache.org --f46d04374a093c6c0404d71d6908 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Yeah, I just checked and the heap size 0.75 warning has been appearing. nodetool info reports: Heap Memory (MB) : 563.88 / 1014.00 Heap Memory (MB) : 646.01 / 1014.00 Heap Memory (MB) : 639.71 / 1014.00 We have plenty of free memory on each instance. Do we need bigger instances or should we just configure each node to have a bigger max heap? On Mon, Mar 4, 2013 at 6:10 PM, Hiller, Dean wrote: > What is nodetool info say for your memory? (we hit that one with memory > near the max and it slowed down our system big time=85still working on > resolving it too). > > Do any logs have the hit 0.75, running compaction OR worse hit 0.85 > running compaction=85.you get that if the above is the case typically. > > Dean > > From: Tom Martin > > Reply-To: "user@cassandra.apache.org" < > user@cassandra.apache.org> > Date: Monday, March 4, 2013 10:31 AM > To: "user@cassandra.apache.org" < > user@cassandra.apache.org> > Subject: Poor read latency > > Hi all, > > We have a small (3 node) cassandra cluster on aws. We have a replication > factor of 3, a read level of local_quorum and are using the ephemeral dis= k. > We're getting pretty poor read performance and quite high read latency i= n > cfstats. For example: > > Column Family: AgentHotel > SSTable count: 4 > Space used (live): 829021175 > Space used (total): 829021175 > Number of Keys (estimate): 2148352 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 0 > Read Count: 67204 > Read Latency: 23.813 ms. > Write Count: 0 > Write Latency: NaN ms. > Pending Tasks: 0 > Bloom Filter False Positives: 50 > Bloom Filter False Ratio: 0.00201 > Bloom Filter Space Used: 7635472 > Compacted row minimum size: 259 > Compacted row maximum size: 4768 > Compacted row mean size: 873 > > For comparison we have a similar set up in another cluster for an old > project (hosted on rackspace) where we're getting sub 1ms read latencies. > We are using multigets on the client (Hector) but are only requesting ~4= 0 > rows per request on average. > > I feel like we should reasonably expect better performance but perhaps I'= m > mistaken. Is there anything super obvious we should be checking out? > > --f46d04374a093c6c0404d71d6908 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Yeah, I just checked and the heap size 0.75 warning has be= en appearing.

nodetool info reports:

Heap Memory (MB) : 563.88 / 1014.00
Heap Memory (MB) : 646.01 / 1014.00
Heap Memory (MB) : 639.71= / 1014.00

We have plenty of free memory= on each instance. =A0Do we need bigger instances or should we just configu= re each node to have a bigger max heap?


O= n Mon, Mar 4, 2013 at 6:10 PM, Hiller, Dean <Dean.Hiller@nrel.gov= > wrote:
What is nodetool info say for your memory? = =A0(we hit that one with memory near the max and it slowed down our system = big time=85still working on resolving it too).

Do any logs have the hit 0.75, running compaction OR worse hit 0.85 running= compaction=85.you get that if the above is the case typically.

Dean

From: Tom Martin <tompoges@gmail.c= om<mailto:tompoges@gmail.com>>
Reply-To: "user@cassandra= .apache.org<mailto:user= @cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Monday, March 4, 2013 10:31 AM
To: "user@cassandra.apach= e.org<mailto:user@cassa= ndra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Poor read latency

Hi all,

We have a small (3 node) cassandra cluster on aws. =A0We have a replication= factor of 3, a read level of local_quorum and are using the ephemeral disk= . =A0We're getting pretty poor read performance and quite high read lat= ency in cfstats. =A0For example:

Column Family: AgentHotel
SSTable count: 4
Space used (live): 829021175
Space used (total): 829021175
Number of Keys (estimate): 2148352
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 67204
Read Latency: 23.813 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Bloom Filter False Positives: 50
Bloom Filter False Ratio: 0.00201
Bloom Filter Space Used: 7635472
Compacted row minimum size: 259
Compacted row maximum size: 4768
Compacted row mean size: 873

For comparison we have a similar set up in another cluster for an old proje= ct (hosted on rackspace) where we're getting sub 1ms read latencies. = =A0We are using multigets on the client (Hector) but are only requesting ~4= 0 rows per request on average.

I feel like we should reasonably expect better performance but perhaps I= 9;m mistaken. =A0Is there anything super obvious we should be checking out?=


--f46d04374a093c6c0404d71d6908--