From user-return-14134-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Thu Dec 09 11:05:01 2010 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 60637 invoked from network); 9 Dec 2010 11:05:01 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 9 Dec 2010 11:05:01 -0000 Received: (qmail 24065 invoked by uid 500); 9 Dec 2010 11:04:59 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 23769 invoked by uid 500); 9 Dec 2010 11:04:58 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 23761 invoked by uid 99); 9 Dec 2010 11:04:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 11:04:58 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of fdmanana@gmail.com designates 209.85.161.54 as permitted sender) Received: from [209.85.161.54] (HELO mail-fx0-f54.google.com) (209.85.161.54) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 11:04:50 +0000 Received: by fxm16 with SMTP id 16so2223765fxm.27 for ; Thu, 09 Dec 2010 03:04:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:content-type:content-transfer-encoding; bh=yRRG1nlneqgQsDAR0TnPgVoBvioUr5GUHOYld/zfnuk=; b=QBuNd+rwtxbKg609S6WDk/DYl5Di7Z+b7C7FsrtFO6Xq2DI0yY08qZuzvrvVaf8pkE DE5j1rhjlc18T3UMNWVThwGNqq7V9vkRzz8cH0X3rn9Bdja7FHjj4gVHwFhLTgtU71Ms Nw16U02VIpyqoUNn0hhxjeUPSSJFz5QEmApaY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; b=M3/qKCt5PkzuuxqO1jvUhORd0So33q1HiHDPh7k9udV31aCc3G9bxpnkVGm4L06Znr eZSMQWJpu/f6hkyAkhKCk+VHPT62mhDCxMUlk0uca7cpH+VO8QH510SYtwgpREc6z2Gh w+0zhMYoD+BmHSzpYUC7QZdj+9kVvqXMLH414= MIME-Version: 1.0 Received: by 10.223.123.209 with SMTP id q17mr3981688far.126.1291892670683; Thu, 09 Dec 2010 03:04:30 -0800 (PST) Sender: fdmanana@gmail.com Received: by 10.223.151.12 with HTTP; Thu, 9 Dec 2010 03:04:30 -0800 (PST) In-Reply-To: <2BC64AB2-0393-4E0C-8D41-842E1B5DA025@netdev.co.uk> References: <52216727-630A-49F6-B919-E691007F1361@netdev.co.uk> <2BC64AB2-0393-4E0C-8D41-842E1B5DA025@netdev.co.uk> Date: Thu, 9 Dec 2010 11:04:30 +0000 X-Google-Sender-Auth: uN5O7thXvuloZNeaXsBWoZ9R03k Message-ID: Subject: Re: Read request throughput From: Filipe David Manana To: user@couchdb.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Hi Huw, Great news! I don't expect you to see any significant performance differences between 1.0.x and 1.1x however. Thanks for letting us know about your tests. best regards, On Thu, Dec 9, 2010 at 10:09 AM, Huw Selley wrote= : > Hi, > > On 8 Dec 2010, at 19:24, Filipe David Manana wrote: > >> Huw, >> >> Today trunk was patched to increase both read and write performance >> when there are several requests in parallel to the same database/view >> index file. > > Great news :) > >> >> The corresponding ticket is https://issues.apache.org/jira/browse/COUCHD= B-980 >> >> Would be much appreciated if you could try the latest trunk and report b= ack :) > > WOW - I built from svn rev 1043651 (again with Erlang R14B) this morning = and have just performed the same jmeter tests with some good results. > I am still seeing the same throughput score from jmeter, ~500 requests/s = but what is interesting is that I can now drive up the threadpool count in = jmeter from 25 (the value I had for my last round of testing) up to 750 wit= h no errors - just increased request latency (which is to be expected). > > Processor utilisation also looks more like I would expect: > > 09:16:43 AM =C2=A0CPU =C2=A0 %user =C2=A0 %nice =C2=A0 =C2=A0%sys %iowait= =C2=A0 =C2=A0%irq =C2=A0 %soft =C2=A0%steal =C2=A0 %idle =C2=A0 =C2=A0intr= /s > 09:16:48 AM =C2=A0all =C2=A0 19.82 =C2=A0 =C2=A00.00 =C2=A0 =C2=A09.19 = =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.04 =C2=A0 =C2=A00.25 =C2=A0 =C2=A00.00 =C2= =A0 70.70 =C2=A0 6136.20 > 09:16:48 AM =C2=A0 =C2=A00 =C2=A0 43.80 =C2=A0 =C2=A00.00 =C2=A0 =C2=A05.= 00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 = =C2=A0 51.20 =C2=A0 1000.20 > 09:16:48 AM =C2=A0 =C2=A01 =C2=A0 38.40 =C2=A0 =C2=A00.00 =C2=A0 21.00 = =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.40 =C2=A0 =C2=A00.00 =C2= =A0 40.20 =C2=A0 =C2=A0460.80 > 09:16:48 AM =C2=A0 =C2=A02 =C2=A0 44.49 =C2=A0 =C2=A00.00 =C2=A0 14.43 = =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2= =A0 41.08 =C2=A0 =C2=A0 =C2=A08.80 > 09:16:48 AM =C2=A0 =C2=A03 =C2=A0 36.87 =C2=A0 =C2=A00.00 =C2=A0 24.05 = =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.40 =C2=A0 =C2=A00.00 =C2= =A0 38.68 =C2=A0 =C2=A0449.80 > 09:16:48 AM =C2=A0 =C2=A04 =C2=A0 45.29 =C2=A0 =C2=A00.00 =C2=A0 11.42 = =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2= =A0 43.29 =C2=A0 =C2=A0 =C2=A00.00 > 09:16:48 AM =C2=A0 =C2=A05 =C2=A0 40.68 =C2=A0 =C2=A00.00 =C2=A0 24.65 = =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2= =A0 34.67 =C2=A0 =C2=A0 =C2=A01.60 > 09:16:48 AM =C2=A0 =C2=A06 =C2=A0 47.20 =C2=A0 =C2=A00.00 =C2=A0 14.40 = =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2= =A0 38.40 =C2=A0 =C2=A0 =C2=A00.00 > 09:16:48 AM =C2=A0 =C2=A07 =C2=A0 13.00 =C2=A0 =C2=A00.00 =C2=A0 10.40 = =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2= =A0 76.60 =C2=A0 =C2=A0 =C2=A00.00 > 09:16:48 AM =C2=A0 =C2=A08 =C2=A0 =C2=A05.01 =C2=A0 =C2=A00.00 =C2=A0 10.= 82 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 = =C2=A0 84.17 =C2=A0 =C2=A0 =C2=A00.00 > 09:16:48 AM =C2=A0 =C2=A09 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2= =A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.40 =C2=A0 =C2=A0= 0.00 =C2=A0 99.60 =C2=A0 =C2=A0455.40 > 09:16:48 AM =C2=A0 10 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.= 00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 = =C2=A0100.00 =C2=A0 =C2=A0 =C2=A00.00 > 09:16:48 AM =C2=A0 11 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.= 00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.20 =C2=A0 =C2=A00.40 =C2=A0 =C2=A00.00 = =C2=A0 99.40 =C2=A0 =C2=A0482.60 > 09:16:48 AM =C2=A0 12 =C2=A0 =C2=A02.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A08.= 58 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 = =C2=A0 89.42 =C2=A0 =C2=A0 10.40 > 09:16:48 AM =C2=A0 13 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.= 00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.40 =C2=A0 =C2=A00.00 = =C2=A0 99.60 =C2=A0 =C2=A0488.80 > 09:16:48 AM =C2=A0 14 =C2=A0 =C2=A00.20 =C2=A0 =C2=A00.00 =C2=A0 =C2=A01.= 40 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.40 =C2=A0 =C2=A00.00 = =C2=A0 98.00 =C2=A0 =C2=A0447.20 > 09:16:48 AM =C2=A0 15 =C2=A0 =C2=A00.60 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.= 80 =C2=A0 =C2=A00.00 =C2=A0 =C2=A00.20 =C2=A0 =C2=A01.20 =C2=A0 =C2=A00.00 = =C2=A0 97.20 =C2=A0 2330.60 > > The box has HyperThreading enabled but the spread of load over cores look= s a lot more like what I would expect to see. > I retried this same test against 1.0.1 with a threadpool count of 750 and= it manages ~126 requests/s with a high error % (again, to be expected), Am= yet to retry against the 1.1.x branch build I was using as it was on the s= ame box the rev 1043651 build it on. > > I have also experimented with increasing the threadpool count against the= rev 1043651 build and I can go upto 950 threads with no errors (throughput= drops a little at that point). This is with a default couch configuration = (only change is delayed_commits=3Dfalse). > > Also, I noticed your relaximation tool (which looks pretty awesome btw) f= rom reading COUCHDB-980 so if I get some time I might give that a run on th= is hardware and see what pretty graphs it can generate :) > > So to summarise: All in all a massive performance win! > > Many thanks to both Adam and Filipe for all your help and advice. It's re= ally nice to see such that couch has such a vibrant community and I hope to= be able to contribute more in the future. > > Regards > Huw --=20 Filipe David Manana, fdmanana@gmail.com, fdmanana@apache.org "Reasonable men adapt themselves to the world. =C2=A0Unreasonable men adapt the world to themselves. =C2=A0That's why all progress depends on unreasonable men."