httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabien COELHO <coe...@cri.ensmp.fr>
Subject Re: Testing/Validation
Date Thu, 07 Jan 1999 09:26:47 GMT

> >> The idea is to test and validate the configuration.
> >> This would include feature testing.
> >> Right now I'm looking for any tools that currently exist.
> >> Even generic tools that are not Apache specific.
> >
> >expect ?
> >wget + diff ?
> 
> That's a poor excuse for a response, Fabien.

Maybe I misunderstood the question. 

> A testing tool has been talked about since 1.0 days, but no one's really
> come up with one.  Most likely it's been because we've relied upon our
> users to do our testing for us... but it would still be nice to have a
> suite that checked some basic functionality before we do a release, like
> Perl's "make test".  A framework for test conditions and results would
> allow one to be written in iterations.  Anyone else interested in this?

I am.

I'm familiar with the validation problem as I'm working on a compiler
project, and participated to the maintainance and extensions of various
in-house automatic non regression test scripts.

I think that the main issue is obviously to make these tests as simple as
possible to write, and then to be fully automatic when they are to be run.

The second issue is to have deterministic output, or to be able to sort
out non deterministic stuff. In the compiler project I'm involved in this
has been a constant issue to avoid validation results to change from one
version to the other. Transposed in the http worlds, an example of the
problems we encountered would be the fact that http headers are not
returned in a deterministic order, because they would be stored in some
internal hash table. Another http-related issue is the parallelism of http
output, and the impact it may have on the output from a test. It may
result in the fact that the test support software would have to remove or
sort part of the tested server output (such as dates...).

The third point that comes of my experience is that we developped a
specific language for our compiler-oriented testing purposes. It looks
like "take this source, set this option, apply this transformation, dump
the intermediate representation, apply this other transformation, aso",
and then we campare the output to the kept reference.

The fourth point I see is that it is uneasy to test some features of
apache in a generic way, as those related to ip number and virtual hosts
for instance. Some support from apache may be useful for these, such as
"ok, even if it is not true, assume you have this ip number just for the
purpose of the test... or assume that / is this directory..."

The fifth general point is that tests must be both positive and
negative. I just mean that errors must be checked, not only stuff that
works. 

Given my previous experience, I would suggest the following as a general
framework: 

 - One directory for each test.

 - In this directory, ONE configuration file for httpd. 
   Maybe the Include feature can be used to have some simple tests.

 - the expected output when the conffile is parsed.

 - maybe a set of directory to serve and other files...

 - A set of http requests to submit (*.in)

 - for each of these the expected output (*.out)
   (simplified? sorted? wirldcards?)

 - some kind of time table (to test parallely/sequentially submitted requests?)
   maybe a perl script with a provided special perl-module could
   do the job...

   ! run_apache conffile other-options
   ! submit_parallel_request 01.in 02.in 03.in
   ! submit_parallel_request 04.in 05.in 06.in
   ! submit_sequential_request 07.in 08.in 09.in

 - a adapted/clever diff would be of some help... (esp. with wildcards)


Fabien.

Mime
View raw message