From dev-return-29291-apmail-httpd-dev-archive=httpd.apache.org@httpd.apache.org Wed May 01 15:11:30 2002 Return-Path: Delivered-To: apmail-httpd-dev-archive@httpd.apache.org Received: (qmail 3812 invoked by uid 500); 1 May 2002 15:11:29 -0000 Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: dev@httpd.apache.org list-help: list-unsubscribe: list-post: Delivered-To: mailing list dev@httpd.apache.org Received: (qmail 3701 invoked from network); 1 May 2002 15:11:29 -0000 X-Curiosity: Killed the Cat X-Huis-aan-Huis-deur-sticker: nee-nee X-Spam: no X-Passed: MX on Gandalf.WebWeaving.org Wed, 1 May 2002 17:05:27 +0200 (CEST) and masked X-No-Spam: Neither the receipients nor the senders email address(s) are to be used for Unsolicited (Commercial) Email without the explicit written consent of either party; as a per-message fee is incurred for inbound and outbound traffic to the originator. Date: Wed, 1 May 2002 17:05:27 +0200 (CEST) From: dirkx@covalent.net X-X-Sender: dirkx@mobile.webweaving.org To: dev@httpd.apache.org Subject: Can AB be compared ? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Just to have some fun - Below is the result of running a build of AB against the same apache 1.3.0 (stock) It is a simple loop - checkout against a tag; cd apache-1.3/src && cp Configuration.tmp && Configuration && ./Configure && make && cd support && make) and then run 20 times ./ab -c 30 -n 10000 and take the last 25 runs. By the way - not a -single- tag failed to build on FreeBSD 5-Current and not a single tag needed a differnet config ! A record hard to beat by commercial software ! For each is displayed to total avaerage time/connection; the 98% confidence interval (all in mSec) and the averegate for that version of AB. Note that typically (with eceptiosn early on) each version of AB produces run within the confidence interval - and that successive versions their confidence intertvals do generally not overlap (except early in the development) in the more recent series. So though this shows that comparing versions of AB from different versions is generally waranted (same mean, same SD) - for recent versions of 1.3 - and that comparisions between version no's is not a good idea. BUT at the same time - that has not always been the case - and that is is kind of a miracle (or a tribute to release management/fact that AB hardly ever changed) that comparison is possible. Also if you include 1.3c and 1.3a then you can conclude that these ranges match - thus from these measurements you would conclude that the code did not change substantially (it may of course do for keep alive or something). Dw. Tag checout #define in ab mean/SD aver CI Tag: APACHE_1_3_24 AB: 1.3d 66 +-3 66.5 +-3 Tag: APACHE_1_3_23 AB: 1.3d 67 +-3 Tag: APACHE_1_3_22 AB: 1.3d 67 +-3 Tag: APACHE_1_3_21 AB: 1.3d 65 +-3 Tag: APACHE_1_3_20 AB: 1.3c 63 +-2 61.8 +-2 Tag: APACHE_1_3_19 AB: 1.3c 60 +-2 Tag: APACHE_1_3_18 AB: 1.3c 61 +-2 Tag: APACHE_1_3_17 AB: 1.3c 61 +-2 Tag: APACHE_1_3_16 AB: 1.3c 63 +-2 Tag: APACHE_1_3_15 AB: 1.3c 62 +-2 Tag: APACHE_1_3_14 AB: 1.3c 63 +-2 Tag: APACHE_1_3_13 AB: 1.3c 62 +-2 Tag: APACHE_1_3_12 AB: 1.3c 62 +-2 Tag: APACHE_1_3_11 AB: 1.3c 60 +-2 Tag: APACHE_1_3_10 AB: 1.3c 63 +-2 Tag: APACHE_1_3_9 AB: 1.3a 60 +-1 60.4 +-1 Tag: APACHE_1_3_8 AB: 1.3a 61 +-1 Tag: APACHE_1_3_7 AB: 1.3a 61 +-1 Tag: APACHE_1_3_6 AB: 1.3 64 +-2 65.2 +-2 Tag: APACHE_1_3_5 AB: 1.3 66 +-3 Tag: APACHE_1_3_4 AB: 1.2 61 +-3 62.9 +-3 Tag: APACHE_1_3_3 AB: 1.2 61 +-4 Tag: APACHE_1_3_2 AB: 1.2 64 +-3 Tag: APACHE_1_3_1 AB: 1.1 60 +-9 65.8 +-10 Tag: APACHE_1_3_0 AB: 1.1 67 +-6 Tag: APACHE_1_3b7 AB: 1.1 64 +-9 Tag: APACHE_1_3b6 AB: 1.1 79 +-8 Dw -- Dirk-Willem van Gulik