Return-Path: 
 <dev-return-19339-apmail-httpd-dev-archive=httpd.apache.org@httpd.apache.org>
Delivered-To: apmail-httpd-dev-archive@httpd.apache.org
Received: (qmail 88599 invoked by uid 500); 23 Aug 2001 15:56:45 -0000
Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
list-help: <mailto:dev-help@httpd.apache.org>
list-unsubscribe: <mailto:dev-unsubscribe@httpd.apache.org>
list-post: <mailto:dev@httpd.apache.org>
Delivered-To: mailing list dev@httpd.apache.org
Received: (qmail 88577 invoked from network); 23 Aug 2001 15:56:45 -0000
Date: Thu, 23 Aug 2001 08:56:20 -0700
From: Justin Erenkrantz <jerenkrantz@ebuilt.com>
To: dev@httpd.apache.org
Subject: Re: Flood
Message-ID: <20010823085620.J17570@ebuilt.com>
References: <A5374D237E78D41195810090279CC91A021EE04D@xcup04.cup.hp.com>
 <20010816101517.O1397@ebuilt.com> <20010823155314.G33621@deejai2.mch.fsc.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20010823155314.G33621@deejai2.mch.fsc.net>;
 from Martin.Kraemer@Fujitsu-Siemens.com on Thu, Aug 23,
 2001 at 03:53:14PM +0200
X-AntiVirus: scanned for viruses by AMaViS 0.2.1-pre3 (http://amavis.org/)
X-Spam-Rating: h31.sny.collab.net 1.6.2 0/1000/N
Status: O
X-Status: 
X-Keywords: 
X-UID: 1096

On Thu, Aug 23, 2001 at 03:53:14PM +0200, Martin Kraemer wrote:
> > 
> > You can devise a user profile that flood will execute (i.e. a sequence
> > of URLs - potentially dynamic) rather than a single static URL as ab
> > lets you do.  -- justin
> 
> How does it compare to httperf?

Well, Aaron, Roy, and myself did look at httperf before we started 
development on flood.  However, you don't have to write C modules
to extend flood's "workload generators" (to use httperf's jargon).  
You can write C modules to extend how flood operates, but that's not
required to specify which URLs you wish to hit.

Basically, we have the ability to mimic a complete user profile in a XML
format.  This snippet is based off of examples/round-robin-example.xml
in httpd-test (slightly cleaned up):

 <urllist>
    <name>Apachelabs.org search profile</name>
    <description>Search for httpd-2.0 commit access and get the first two messages in the thread</description>
    <url method="POST" payload="method=and&amp;format=builtin-long&amp;sort=score&amp;config=htdig&amp;restrict=&amp;exclude=&amp;words=httpd-2.0+commit+access"
responsetemplate="&lt;a href=&quot;([^&quot;]*)&quot;&gt;" responsename="id">http://www.apachelabs.org/cgi-bin/htsearch</url>
    <url requesttemplate="${id}" responsetemplate="&lt;A HREF=&quot;([^&quot;]*)&quot;&gt;Next&lt;/A&gt;" responsename="next" />
    <url requesttemplate="http://www.apachelabs.org${next}" />
  </urllist>

The first URL does a POST as a htdig query searching for "httpd-2.0
commit access" on apachelabs.org.  Notice the HTML escaping present -
since we are using XML format, we must escape everything - that kind of
sucks, but without writing our own pseudo-XML parser (we use APR-util's 
XML API) we don't have much of a choice here.  =)

When the search returns, we look for the first occurrance of "<a
href="http://www.apachelabs.org/foo/bar.html"> (regexp matching) and 
place http://www.apachelabs.org/foo/bar.html in the variable id.  

The next URL simply uses that variable directly to be the next URL to
hit.  Then, once we retrieve that URL, we look for 
"<A HREF="/spaz/bar.html">Next</A> and store /spaz/bar.html into the 
variable next.

We then hit the final URL in our sequence with that message.

See if you can do this in three lines with httperf.  =)

Current features that httperf supports that we don't:
Chunked encoding (on our short list when we get around to it)
Log-file replay (this is conceptually easy except for POST payloads)
As much network information detail

Flood's output is pretty raw at this point.  We output it so that it is
easy to run your own statistical analysis/summary on the results.  So,
we expect you to write awk/perl scripts - we have a bunch that we use
internally to generate result summaries.  I may commit modified versions
of some of them when time permits.  (We also run flood in combination
with sar and snoop at times to generate more OS specific data...)

Features we support that httperf doesn't:
SSL
Better delay handling (per URL)
slightly-easier configuration
Threading

Threading is a mixed bag performance-wise as poll() based is slightly 
better performance wise, but remember that we aren't going for 
meaningless benchmarks *choke* ab *choke*, but "real-world" numbers - 
so a thread/user makes a lot of sense implementation-wise.

Like httperf, we'd like to have a "daemon-mode" where we can spread out
flood sessions across multiple machines and have all of the reports go
to one central place.

Anyway, we're using flood internally at eBuilt to performance test 
one of our client's websites (the XML gets quite hairy as the site is 
very complex).  So, we're trying to first build in the features
that we need and then focus on other things later as time permits.
We've found a number of bottlenecks and problematic pages under load
on that site.  We're able to tell them which pages are worse under
load, etc, etc.  So, it is scratching our itch.

Ideally, other people will find this tool of value and use it.  And,
if they find problems (or need new features), they can submit patches
(or commit themselves).  We'd definitely welcome the help.  HTH.  
-- justin