gump-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kentb" <ke...@earthlink.net>
Subject RE: Test data wanted
Date Sat, 13 Sep 2008 14:38:32 GMT
Stefan,

I plotted the runtimes of all the test cases last night and the graph looked
really strange. Sometime in the night I figured out that the data was in
seconds, not milliseconds. Scaling by 1000 provides more or less the picture
I expected: http://tinyurl.com/4r5yxs. The test runtimes follow a power law
distribution (perhaps not to a statistician, but it looks like a straight
line to me). I have no idea why this is true, but it's consistent with other
corpi of test runs I've analyzed. This is just so cool--thanks for your
help.

I'll keep you posted as I analyze more data. If you can think of other
repositories of test runtimes, I'd appreciate a pointer.

Cheers,

Kent

-----Original Message-----
From: Stefan Bodewig [mailto:bodewig@apache.org] 
Sent: Friday, September 12, 2008 6:26 AM
To: kentb@earthlink.net
Cc: general@gump.apache.org
Subject: Re: Test data wanted

On Wed, 10 Sep 2008, <kentb@earthlink.net> wrote:

> Good to hear from you again. Thank you for your kind offer of help. 
> Given the time I have to give to the task and the rustiness of my Unix 
> skills, I suspect it would take as much elapsed time for me to ferret 
> out the files as it would for you to gather them.

It turned out that Ant was pretty well suited for the task,

  <zip destfile="${target}/testresults.zip">
    <fileset dir="${gump.base}" includes="**/*.xml" excludes="cvs/">
      <and>
        <readable/>
        <contains text="&lt;testsuite"/>
      </and>
    </fileset>
  </zip>

and about ten minutes of waiting was really all it took to collect all XML
files that contain "<testsuite".  There will certainly be false positives in
the zip like Ant's own AntUnit tests (<antunit> creates an XML output
similar to <junit>) but I'm sure you can extract those.

> When you get a chance, I'd appreciate it if you could put them 
> together for me in whatever format is easy for you to create.

vmgump's webserver doesn't serve home dirs anymore and I don't want to
fiddle with it right now, so I've moved the zip over to my place (about
8MB).  Grab it from <http://stefan.samaflost.de/code/testresults.zip> and
I'll delete it some time later over the next days.

> Just as an introduction, one of the graphs I created shows the number 
> of tests per test suite and the number of failures and errors per test 
> suite.

JUnit's notion of a suite and Ant's are not identical, this stems from the
way Thomas Haas and I used JUnit when we wrote the <junit> task in May/June
2000 and has carried over from there.

Both of us didn't use explicit TestSuites but only classes inheriting from
TestCase and this is Ant's expectation - a <testuite> in Ant's terms is
whatever invoking the static suite() method or automatically extracting all
test methods of a single class yield (or more recently what you get by
wrapping the test class in a JUnit4Adapter).  You get exactly one
<testsuite> for each class the TestRunner has been invoked on.

> (The x axis is roughly 2^n.) The astonishing thing for me in this case 
> is the fairly clear power law distribution in the number of failures. 
> Why in the world would that be?

You'd probably need to see how data evolves over time to really see what
happens here.  Are the few cases with many failures simply tests that are
known to fail and that get ignored over time?  Or some sort of refactoring
that left failing tests behind with no time to adapt them?

Stefan


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org


Mime
View raw message