From user-return-12929-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Fri Feb 04 19:44:34 2011 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 93520 invoked from network); 4 Feb 2011 19:44:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 4 Feb 2011 19:44:34 -0000 Received: (qmail 71497 invoked by uid 500); 4 Feb 2011 19:44:32 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 71449 invoked by uid 500); 4 Feb 2011 19:44:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 71441 invoked by uid 99); 4 Feb 2011 19:44:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Feb 2011 19:44:31 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_FROM,FREEMAIL_REPLY,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dmdabbs@gmail.com designates 209.85.214.172 as permitted sender) Received: from [209.85.214.172] (HELO mail-iw0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Feb 2011 19:44:24 +0000 Received: by iwc10 with SMTP id 10so2738941iwc.31 for ; Fri, 04 Feb 2011 11:44:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:to:references:in-reply-to:subject:date :message-id:mime-version:content-type:content-transfer-encoding :x-mailer:thread-index:content-language; bh=8lgHBmCEMPNXgUqHCdZpKBCahFzNXcbvHIux2JLcfsg=; b=BJ8U0TOJliI0vrtULEFn77eOoPNccfzVv9eGq6TdZQV8yW+OW4y0ff374UUtEF2v81 YbZyXAMwa9fs9YbXU1y0xnI6Y3Vc8/35UewDIxEOkYAVisAgQ5LfZtAW1U1f067Avj+f dWj/YpCzAjK+IzeQ3MdRbOC/R/VOhOgcN5w6g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:references:in-reply-to:subject:date:message-id:mime-version :content-type:content-transfer-encoding:x-mailer:thread-index :content-language; b=JoW/FF8m2yQ8vVmhC04RdkiG0Nn5MWh8rPLMqg+/jMtRAvczLLqbZqwi63JNflZvXQ JDbifH1cEwYuzKV5qp0ye8eNilW4y5hqyGqIsVEjcCczUruyQNdq6hZjB9fcaPgGuGb7 MUdMoIA3YJlNvChdU5OI83YBdDvKaQYI9CBIo= Received: by 10.42.165.7 with SMTP id i7mr5396445icy.452.1296848643345; Fri, 04 Feb 2011 11:44:03 -0800 (PST) Received: from DTILDDABBS ([66.252.72.218]) by mx.google.com with ESMTPS id k42sm776618ick.20.2011.02.04.11.44.01 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 04 Feb 2011 11:44:02 -0800 (PST) From: "David Dabbs" To: References: <018e01cbc364$28c86a00$7a593e00$@com> In-Reply-To: Subject: RE: Tracking down read latency Date: Fri, 4 Feb 2011 13:44:01 -0600 Message-ID: <008401cbc4a3$dfacfac0$9f06f040$@com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: AcvDw5QLI4/lVH5dTZKC9sdb7tWAVAA3tZgA Content-Language: en-us Thank you both for your advice. See my updated iostats below. >From: sridhar.basam@gmail.com [mailto:sridhar.basam@gmail.com] On Behalf Of sridhar basam >Sent: Thursday, February 03, 2011 10:58 AM >To: user@cassandra.apache.org >Subject: Re: Tracking down read latency > >The data provided is also a average value since boot time. Run the -x as suggested below but run it via a interval of around 5 seconds. You very well could be having i/o issue, >it is hard to tell from the overall average value you provided. Collect "iostat -x 5" during the times when you see slow reads and see how busy the disks are. >Sridhar >On Thu, Feb 3, 2011 at 3:21 AM, Peter Schuller wrote: >> $ iostat > >As rcoli already mentioned you don't seen to have an I/O problem, but >as a point of general recommendation: When determining whether you are >blocking on disk I/O, pretty much *always* use "iostat -x" rather than >the much less useful default mode of iostat. The %util and queue >wait/average time columns are massively useful/important; without them >one is much more blind as to whether or not storage devices are >actually saturated. > > Peter Schuller Our data is on sdb, commit logs on sdc. So do I read this correctly that we're 'await'ing 6+millis on average for data drive (sdb) requests to be serviced? $iostat -x 5 avg-cpu: %user %nice %system %iowait %steal %idle 0.59 0.00 0.22 0.94 0.00 98.25 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 11.20 0.00 42.00 0.00 4993.60 0.00 118.90 0.28 6.77 5.22 21.92 sdb1 11.20 0.00 42.00 0.00 4993.60 0.00 118.90 0.28 6.77 5.22 21.92 sdc 0.00 31.00 0.00 1.40 0.00 259.20 185.14 0.00 0.14 0.14 0.02 sdc1 0.00 31.00 0.00 1.40 0.00 259.20 185.14 0.00 0.14 0.14 0.02 avg-cpu: %user %nice %system %iowait %steal %idle 0.56 0.00 0.18 1.08 0.00 98.17 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 8.80 0.00 49.40 0.00 5936.00 0.00 120.16 0.33 6.62 5.22 25.78 sdb1 8.80 0.00 49.40 0.00 5936.00 0.00 120.16 0.33 6.62 5.22 25.78 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.99 0.00 0.22 1.08 0.00 97.71 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 11.40 0.00 46.20 0.00 5147.20 0.00 111.41 0.30 6.55 5.58 25.80 sdb1 11.40 0.00 46.20 0.00 5147.20 0.00 111.41 0.30 6.55 5.58 25.80 sdc 0.00 7.40 0.00 0.80 0.00 65.60 82.00 0.00 0.25 0.25 0.02 sdc1 0.00 7.40 0.00 0.80 0.00 65.60 82.00 0.00 0.25 0.25 0.02 avg-cpu: %user %nice %system %iowait %steal %idle 0.68 0.00 0.23 0.95 0.00 98.13 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.80 0.00 0.80 0.00 12.77 16.00 0.00 0.25 0.25 0.02 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.80 0.00 0.80 0.00 12.77 16.00 0.00 0.25 0.25 0.02 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 5.19 0.00 38.12 0.00 4356.09 0.00 114.26 0.26 6.70 5.91 22.53 sdb1 5.19 0.00 38.12 0.00 4356.09 0.00 114.26 0.26 6.70 5.91 22.53 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00