From cassandra-user-return-1624-apmail-incubator-cassandra-user-archive=incubator.apache.org@incubator.apache.org Thu Dec 03 22:59:35 2009 Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 82132 invoked from network); 3 Dec 2009 22:59:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 3 Dec 2009 22:59:35 -0000 Received: (qmail 42663 invoked by uid 500); 3 Dec 2009 22:59:34 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 42633 invoked by uid 500); 3 Dec 2009 22:59:34 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 42624 invoked by uid 99); 3 Dec 2009 22:59:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 22:59:34 +0000 X-ASF-Spam-Status: No, hits=-6.4 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_MED X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [15.201.24.20] (HELO g4t0017.houston.hp.com) (15.201.24.20) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 22:59:32 +0000 Received: from G6W0640.americas.hpqcorp.net (g6w0640.atlanta.hp.com [16.230.34.76]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by g4t0017.houston.hp.com (Postfix) with ESMTPS id 4F72D3835C for ; Thu, 3 Dec 2009 22:59:11 +0000 (UTC) Received: from G4W1852.americas.hpqcorp.net (16.234.97.230) by G6W0640.americas.hpqcorp.net (16.230.34.76) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 3 Dec 2009 22:58:43 +0000 Received: from GVW0432EXB.americas.hpqcorp.net ([16.234.32.146]) by G4W1852.americas.hpqcorp.net ([16.234.97.230]) with mapi; Thu, 3 Dec 2009 22:58:43 +0000 From: "Freeman, Tim" To: "cassandra-user@incubator.apache.org" Date: Thu, 3 Dec 2009 22:58:41 +0000 Subject: RE: Persistently increasing read latency Thread-Topic: Persistently increasing read latency Thread-Index: Acp0alrQ3YDNNoDiShapSGmx/WQzcQAALKHw Message-ID: <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850C2317@GVW0432EXB.americas.hpqcorp.net> References: <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850190C6@GVW0432EXB.americas.hpqcorp.net> <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850C20C5@GVW0432EXB.americas.hpqcorp.net> <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850C20F4@GVW0432EXB.americas.hpqcorp.net> <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850C228B@GVW0432EXB.americas.hpqcorp.net> <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850C22D8@GVW0432EXB.americas.hpqcorp.net> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 >So this is working as designed, but the design is poor because it >causes confusion. If you can open a ticket for this that would be >great. Done, see: https://issues.apache.org/jira/browse/CASSANDRA-599 >What does iostat -x 10 (for instance) say about the disk activity? rkB/s is consistently high, and wkB/s varies. This is a typical entry with= wkB/s at the high end of its range: >avg-cpu: %user %nice %sys %iowait %idle > 1.52 0.00 1.70 27.49 69.28 > >Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avg= rq-sz avgqu-sz await svctm %util >sda 3.10 3249.25 124.08 29.67 26299.30 26288.11 13149.65 13144.06= 342.04 17.75 92.25 5.98 91.92 >sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 = 0.00 0.00 0.00 0.00 0.00 >sda2 3.10 3249.25 124.08 29.67 26299.30 26288.11 13149.65 13144.06= 342.04 17.75 92.25 5.98 91.92 >sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 = 0.00 0.00 0.00 0.00 0.00 and at the low end: >avg-cpu: %user %nice %sys %iowait %idle > 1.50 0.00 1.77 25.80 70.93 > >Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avg= rq-sz avgqu-sz await svctm %util >sda 3.40 817.10 128.60 17.70 27828.80 6600.00 13914.40 3300.00 = 235.33 6.13 56.63 6.21 90.81 >sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 = 0.00 0.00 0.00 0.00 0.00 >sda2 3.40 817.10 128.60 17.70 27828.80 6600.00 13914.40 3300.00 = 235.33 6.13 56.63 6.21 90.81 >sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 = 0.00 0.00 0.00 0.00 0.00 Tim Freeman Email: tim.freeman@hp.com Desk in Palo Alto: (650) 857-2581 Home: (408) 774-1298 Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and Thur= sday; call my desk instead.) -----Original Message----- From: Jonathan Ellis [mailto:jbellis@gmail.com]=20 Sent: Thursday, December 03, 2009 2:45 PM To: cassandra-user@incubator.apache.org Subject: Re: Persistently increasing read latency On Thu, Dec 3, 2009 at 4:34 PM, Freeman, Tim wrote: >>Can you tell if the system is i/o or cpu bound during compaction? > > It's I/O bound. =A0It's using ~9% of 1 of 4 cores as I watch it, and all = it's doing right now is compactions. What does iostat -x 10 (for instance) say about the disk activity?