Return-Path: 
 <cocoon-dev-return-17194-apmail-xml-cocoon-dev-archive=xml.apache.org@xml.apache.org>
Delivered-To: apmail-xml-cocoon-dev-archive@xml.apache.org
Received: (qmail 84799 invoked by uid 500); 23 Sep 2001 19:35:11 -0000
Mailing-List: contact cocoon-dev-help@xml.apache.org; run by ezmlm
Precedence: bulk
list-help: <mailto:cocoon-dev-help@xml.apache.org>
list-unsubscribe: <mailto:cocoon-dev-unsubscribe@xml.apache.org>
list-post: <mailto:cocoon-dev@xml.apache.org>
Reply-To: cocoon-dev@xml.apache.org
Delivered-To: mailing list cocoon-dev@xml.apache.org
Received: (qmail 84784 invoked from network); 23 Sep 2001 19:35:11 -0000
Message-Id: <200109231934.f8NJYYV10916@localhost.localdomain>
X-Mailer: exmh 2.5 07/16/2001 with XEmacs 21.4.4 on Linux 2.4.9
From: Ovidiu Predescu <ovidiu@cup.hp.com>
To: Jeff Turner <jeff@socialchange.net.au>
Cc: cocoon-dev@xml.apache.org, tec@orion.rgv.hp.com
Subject: Re: testing/performance framework for Cocoon 
In-Reply-To: Your message of "Sun, 23 Sep 2001 13:53:58 +1000."
             <20010923135358.A22462@kermit.socialchange.net.au>
X-Url: http://www.geocities.com/SiliconValley/Monitor/7464/
X-Image-Url: http://www.geocities.com/SiliconValley/Monitor/7464/ovidiu.tiff
X-Face: 
 ?(@Y~qjBA}~8ZMh5gM4{Q{bE_*:sCJ3@Z?{B*Co=J!#8bb~-z?-0._vJjt~MM59!MjxG%>U
 5>MW^2-\7~z04buszR^=m^U|m66>FdR@cFwhb;.A(8*D.QmLkK]z,md0'HiOE\pyeiv_PACR+P:Cm.
 wq_%l':E:q]g-UCc>r&s@BVo'kFN;(\9PF22Myg5w%nUBWQ6MJJ#qL#w>2oxckP'H:\$9F"mxsz]Dg
 k{1`fTcP'Y$CgGnc^paTV$dzhVX<SVN1RPb="gz3>+;(U$;Eb)P<>G)g)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Date: Sun, 23 Sep 2001 12:34:34 -0700
Sender: ovidiu@cup.hp.com
X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N

Jeff,

Thanks for your thoughtfull insights, I really appreciate them.

On Sun, 23 Sep 2001 13:53:58 +1000, Jeff Turner <jeff@socialchange.net.au> wrote:

> I recently required something like this. I first played around with Tomcat's
> Ant task for doing unit testing (see
> jakarta-tomcat-4.0/tester/src/bin/tester.xml). It's used like this:
> 
> <project name="Tester" default="all">
>   <taskdef  name="tester"     classname="org.apache.tester.TestClient"/>
>   ..
>   <tester host="${host}" port="${port}" protocol="${protocol}"
>            request="/index.html" debug="${debug}"
>            status="200"/>
>   <tester host="${host}" port="${port}" protocol="${protocol}"
>          request="${context.path}/WrappedGolden01"
>          golden="${golden.path}/Golden01.txt"/>
>   <tester host="${host}" port="${port}" protocol="${protocol}"
>           debug="${debug}"
>           request="${context.path}/protected/Authentication01"
>           inHeaders="Authorization:Basic dG9tY2F0OnRvbWNhdA=="
>   ..
> 
> However, this system is rather monolithic and hard to extend. All new tests
> would have to be added to TestClient directly.

I looked at TestClient, and it seems more oriented towards low level
HTTP testing.

> I had a brief poke at httpunit, but it seemed to be more oriented at HTML
> testing.

I agree with you here, httpunit is very heavily HTML
oriented. Nevertheless some people will need this kind of
functionality, thus whatever framework we come up with should allow
the usage of httpunit.

> So I'm now playing with a testing tool called Latka, developed by Morgan
> Delagrange in jakarta-commons/latka.
> 
>   "Latka is a functional (end-to-end) testing tool. It is implemented in Java,
>   and uses an XML syntax to define a series of HTTP (or HTTPS) requests and a
>   set of validations used to verify that the request was processed correctly."
> 
>   -- http://jakarta.apache.org/commons/latka/index.html

Thanks for the pointer, I took a look at it.

> It has quite a few validation techniques already, but none specific to XML.
> However, it's very easy to write new validators. I had a MIME type validator
> (eg check for text/xml) and an XML validation validator going in a few hours. I
> expect it would be quite trivial to implement an XPath test:
> 
> > <check> - checks an XPath in the XML document returned by an <url-get>
> > or <url-post>.

The XPath validation could happen using Jaxen (http://www.jaxen.org).

> Here's a sample XML file, doing a few tests against XML services:
> 
> <?xml version="1.0" standalone="no"?>
> <!DOCTYPE suite SYSTEM "../conf/suite.dtd">
> <suite defaultHost="newgate.socialchange.net.au" label="newgate">
> 
>     <request path="/sightings/servlet/WFSSightingsServlet" label="WFSSightings">
>         <validate>
>             <statusCode label="valid HTTP status code"  code="200"/>
>             <mimeType label="Correct XML MIME type" value="text/xml"/>
>         </validate>
>     </request>
> 
>     <request host="clearinghouse1.fgdc.gov"
>         path="/scripts/ogc/ms.pl?WMTVER=1.0&amp;request=CAPABILITIES"
>         label="SCO WMS">
>         <validate>
>             <mimeType value="text/xml"/>
>             <xmlValidator/>
>         </validate>
>     </request>
> </suite>
> 
> (David Crossley and other GIS people will understand those tests;)

This is part of the functionality I'm looking for. I also need some
rudimentary support for defining variables, so that values extracted
from one response document can be used later in the test. Ant
properties seem to do a good job of this.

> Note that it doesn't use Ant. I suspect this is actually a good
> thing. Ant 1.x is not flexible enough to be easily repurposed as a
> http testing engine. This will apparently be fixed in Ant 2, which
> will be a generic "task execution engine", but for now, it's
> probably harder to turn Ant into something it's not, than to start
> again. For instance, there have been *many* submissions of an
> iterate, or <foreach> task:

As I looked at Ant, it appears to me the internal architecture is well
designed and could support the testing framework I think of. The only
problem I see is the usage of SAX1 API, which precludes one from using
XML namespaces. I think this problem can be fixed though, by doing the
appropriate changes within Ant, the code is very well localized. Other
than this issue, I don't see any reason why Ant would not be a good
framework to support this testing framework.

Would you please elaborate on your reasons why Ant's engine would not
provide a good testing framework?

> > <iterate> - creates a "for" loop so we can do very simple loops
> 
> But the ant-dev team are very much against turning Ant into an XML scripting
> language, and reject such tasks.

The Ant based testing framework can be very well an optional package,
not part of the official distribution. The only concern I have is with
changes to the Ant internals, needed to support this framework. I
would certainly want to have them included back in the official
release. But the testing framework can very well live outside Ant.

> That thought leads to one serious suggestion: how about implementing
> the tests in Python, and then using a Python-to-Java compiler
> (http://jython.sourceforge.net/) to generate bytecode?

I have done some extensive Python programming in the past, although
not much JPython. I'm not looking at a full programming language to
implement this framework. If we go on this path, why not use Java
directly, why use yet another language the developers/testers need to
learn?

I'm looking at a high level language in which tests could be expressed
easily. I believe an XML-based language, implemented on top of Ant's
framework, provide a good starting point. Even if we use
Python/JPython we would end up writing a lot of supporting code, and
I'm not sure you have any advantages over Java, as both languages have
very similar concepts.

> > <spawn> - creates a given number of threads and executes the tasks
> > specified as children. This could be used to simulate multiple clients
> > connecting to a URL.
> 
> It may be possible to hack Ant 1.4's <parallel> task to do this sort
> of thing.  However, the ant-dev committers would be unlikely to
> accept your addition, since it amounts to an <iterate> task.

Yes, the <spawn> task would probably implemented as an extension to
the <parallel> task. It differs however in the way it
works. <parallel> spawns multiple threads, each one runnning a child
task of <parallel>. <spawn> runs all of its child tasks in a
sequential order, but spawns multiple threads that run the exact same
sequence.

> So I think the options (in order of my preference;) are:
>  - use and extend Latka to do the testing (lots of work)
>  - write tests in python, compile to java (less work)
>  - turn Ant into testing tool with testing-specific tasks
>  - use httpunit or another framework

I believe that no matter what route we go, we should not exclude other
options from being used. If somebody feels httpunit is the framework
to use to test a particular feature, it should be possible to use it.

The way I see this implemented is by allowing tests written with
multiple methodologies. All the tests however will have to be driven
by a common Ant build.xml, so that the interface provided to the user
is always uniform, e.g.

./build.sh test


Regards,
-- 
Ovidiu Predescu <ovidiu@cup.hp.com>
http://orion.nsr.hp.com/ (inside HP's firewall only)
http://sourceforge.net/users/ovidiu/ (my SourceForge page)
http://www.geocities.com/SiliconValley/Monitor/7464/ (GNU, Emacs, other stuff)

---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org