ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Ant Wiki] Update of "Proposals/EnhancedTestReports" by SteveLoughran
Date Mon, 03 Dec 2007 14:33:58 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Ant Wiki" for change notification.

The following page has been changed by SteveLoughran:
http://wiki.apache.org/ant/Proposals/EnhancedTestReports

The comment on the change is:
Look in detail at how badly XML-in-Exception.toString() get's processed 

------------------------------------------------------------------------------
   * No way to attach artifacts such as VMWare images to test results
   * Only one failure is allowed per test case.  (JUnit requires there be only one failure
per test case, but other test frameworks e.g. Selenium allow for the possibility of multiple
failures per test case.)
  
- 
  Summary: it was good at the time, but as testing has got more advanced, we need to evolve
the format (carefully)
  
  == Radical Alternatives ==
@@ -94, +93 @@

  
  === Improved Logging information ===
  
-  * Retain the current stdout+stderr logs, but also allow people to bind to custom log4j/commons-logging/java.util.logging
back ends that grab the raw events and log them as structured XML events (host,process,thread,level,timestamp,text).
These would be renderable at different levels. This is not something we'd do automatically
as it can change application behaviour. We'd have to provide the custom back-ends for the
loggers and offer a switch to turn this on in the task; the switch would set the properties
for the forked process (and it would have to be forked) to log through our system. There's
an
+  * Retain the current stdout+stderr logs, but also allow people to bind to custom log4j/commons-logging/java.util.logging
back ends that grab the raw events and log them as structured XML events (host,process,thread,level,timestamp,text).
These would be renderable at different levels. This is not something we'd do automatically
as it can change application behaviour. We'd have to provide the custom back-ends for the
loggers and offer a switch to turn this on in the task; the switch would set the properties
for the forked process (and it would have to be forked) to log through our system. There's
an 
- [http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/xunit/src/org/smartfrog/services/xunit/serial/LogEntry.java?view=markup
example class] in the smartfrog repository to capture log entries [http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/xunit/src/org/smartfrog/services/xunit/serial/ThrowableTraceInfo.java?view=markup
and accompanying exceptions] that can be captured serialized and shared between processes.

+ [http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/xunit/src/org/smartfrog/services/xunit/serial/LogEntry.java?view=markup
example ] in the smartfrog repository to capture log entries and accompanying[http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/xunit/src/org/smartfrog/services/xunit/serial/ThrowableTraceInfo.java?view=markup
 exceptions] that can be captured serialized and shared between processes. 
  
   * XSL style sheets to merge output, ordering by clock (received time) and displaying log
messages from different machines/levels in different colour
   * easy turn on/off display of different levels in viewers; maybe the test runner itself,
though log4j.properties and similar can do this (actually, given that java.util logging is
such a PITA to set up, we could add helper operations in the <java> task for all to
use.
@@ -141, +140 @@

  
  MattDoar: I like the idea of being able to define my own test states, but I would like to
see perhaps half a dozen common ones predefined. Perhaps:
  
- * Pass
+  * Pass
- * Fail
+  * Fail
- * Error - unable to determine pass or fail, which includes timeout
+  * Error - unable to determine pass or fail, which includes timeout
- * Skipped
+  * Skipped
- * In Progress, aka not-yet-finished
+  * In Progress, aka not-yet-finished
  
  SteveLoughran: I like failure itself to be categorised. So every state can be a pass state,
a warning or a failure. Warnings could include "passing, but took 50% longer than usual".
Failures can include "tests passed but outside allowed time" "tests passed but memory consumption
was over the limit", as well as simple "some assertion failed". Maybe we decouple result name
from outcome, with a limited set of outcomes, but an unlimited set of outcome categories.

  
@@ -197, +196 @@

  
  What would this look like? Perhaps and element with URLs to related issues?
  
- = Examples =
+ == Examples of existing tests ==
+ 
+ === multi-site tests ===
+ 
+ http://people.apache.org/~stevel/cddlm/interop/ current presentation of multi-host interop
tests]. This directory contains the results of running the same client against three SOAP
endpoints, implementing the set of tests agreed by a standard's working group. 
+  1. Some of the tests fail 'unimplemented' meaning they haven't been unimplemented; these
should really be warnings or skipped tests whose absence is noted, and which all would have
the tag 'unimplemented'.
+  1. The data was collected on a single machine, 3 separate JVMs each with a different endpoint
property set to test against a different remote endpoint.
+  1. The tests throw faults whose toString() method return the SOAPFault. This is being embedded
into the error text unescaped. For example, in [http://people.apache.org/~stevel/cddlm/interop/ourgrid/org/smartfrog/services/deployapi/test/system/alpine/deployapi/api/creation/0_Api_05_system_create_destroy_Test.html
one test] the output appears as
+ {{{
+ ns1:Server
+ Server Error
+ 
+ Server
+ Error at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:473)
+ }}}
+ If you look at the source, it is more informative, showing that the XSL didnt strip the
?xml declaration, or escape any XML
+ {{{
+ <code>
+  <?xml version="1.0" encoding="UTF-8"?><br/>
+  <Fault xmlns="http://schemas.xmlsoap.org/soap/envelope/"><br/>
+   <faultcode xmlns:ns1="http://xml.apache.org/axis/" xmlns="">ns1:Server</faultcode><br/>
+   <faultstring xmlns="">Server Error</faultstring><br/>
+   <detail xmlns=""><br/>
+     <stackTrace xmlns:ns2="http://xml.apache.org/axis/" xmlns="http://xml.apache.org/axis/">Server<br/>
+       Error at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:473)<br/>
+       at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)<br/>
+       at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)<br/>
+       at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) <br/> 
+       at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)<br/>
+       at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) <br/> 
+       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)<br/>
+       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)<br/>
+       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)<br/>
+       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)<br/>

+       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)<br/>
+       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)<br/>
+       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)<br/>
+       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)<br/>
+       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)<br/>
+       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler
+          .processConnection(Http11Protocol.java:744)<br/>
+       at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)<br/>
+       at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)<br/>
+       at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)<br/>
+       at java.lang.Thread.run(Thread.java:595) </stackTrace><br/>
+     <hostname xmlns:ns3="http://xml.apache.org/axis/" xmlns="http://xml.apache.org/axis/">cddlm</hostname><br/>
+   </detail><br/>
+ </Fault><br/>
+ <br/><br/>
+ 	at org.smartfrog.projects.alpine.transport.http.HttpTransmitter.transmit(HttpTransmitter.java:184)<br/>
+         at org.smartfrog.projects.alpine.transport.Transmission.call(Transmission.java:128)<br/>
+ 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)<br/>
+ 	at java.util.concurrent.FutureTask.run(FutureTask.java:123)<br/>
+ 	at org.smartfrog.projects.alpine.transport.DirectExecutor.execute(DirectExecutor.java:32)<br/>
+ 	at org.smartfrog.projects.alpine.transport.TransmitQueue.transmit(TransmitQueue.java:106)<br/>
+ 	at org.smartfrog.projects.alpine.transport.Session.queue(Session.java:204)<br/>
+ 	at org.smartfrog.services.deployapi.alpineclient.model.PortalSession.beginCreate(PortalSession.java:109)<br/>
+ 	at org.smartfrog.services.deployapi.alpineclient.model.PortalSession.create(PortalSession.java:138)<br/>
+ 	at org.smartfrog.services.deployapi.test.system.alpine.deployapi.api.creation.Api_05_system_create_destroy_Test.
+             testCreateDestroySystem(Api_05_system_create_destroy_Test.java:39)<br/>
+ </code>
+ }}}
+ 
+ What we are seeing here, then, is a Java Exception that contains a SOAPFault raised by Axis1.x
at the far end of an HTTP link, but most of this is lost in the textual report. While adding
custom XSL for SOAPFaults is probably out (esp. because JAX-WS 2.0 always tries to hide them),
we should present the returned XML as (a) it's informative and (b) what we do now is a security
risk. What if an Exception.toString() generated HTML with script tags?
+ 
+ 
+ 
+ 
+ = Examples of Extended tests =
  
  Examples of what a new Test Report could look like.
  

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org


Mime
View raw message