incubator-stdcxx-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Sebor <se...@roguewave.com>
Subject Re: RFC: publishing nightly test results
Date Tue, 14 Nov 2006 19:27:16 GMT
Farid Zaripov wrote:
>>-----Original Message-----
>>From: Martin Sebor [mailto:sebor@roguewave.com] 
>>Sent: Monday, November 13, 2006 6:53 PM
>>To: stdcxx-dev@incubator.apache.org
>>Subject: Re: RFC: publishing nightly test results
>>
>>Any comments or questions on this? Farid?
> 
> 
>   Not yet. I found it useful as for me.
> 
>   I see "DATA" in State column in rows with results on MSVC. The tooltop
> comment says
> "Incomplete data". Can I help with this? The windows build
> infrastructure produces the build
> log (i.e. msvc-7.1-15s.html) and perhaps this log can be attached? The
> size of this log file
> is big due to HTML format and UNICODE characters (on MSVC8). We can
> modify
> makelog.wsf script to produce the ASCII text log file as well as html
> one.

Andrew and I brainstormed about how to make it possible and easy
for others (such as yourself) to submit stdcxx test results so that
they could be published along with ours. I think we came up with a
pretty nifty idea. Here's how it would work:

1. Create a script, say stdcxx-test, that when executed will pull
    down the latest stdcxx sources from svn (or perhaps use the
    already downloaded sources instead), run one or more builds
    (perhaps all possible builds by default) with a given compiler,
    collecting the logs and results in the currently published form
    (i.e., config.h, config.log, and build.log). Also have the script
    compute the total number of warnings, tests and failed tests,
    examples and failed examples, etc. When done with the builds,
    have the script tar and gzip the logs into one tarball per build
    and make each available to the world under a URL. Then send an
    email (e.g., to stdcxx-dev) in a preset format outlining the
    results of each build (each email could contain one or more
    results). Something like:

      Result: http://tinyurl.com/yfsrzf
      OS: AIX 5.2                  # any string
      Compiler: VisuaAge C++ 6.0   # any string
      Build Type: 11d              # valid build type
      Timestamp: 11/12 03:46       # date/time (optional)
      Warnings: 127                # number of warnings
      Examples: 11/132             # failed/total examples
      Tests: 38/121                # failed/total tests
      Locales: 156/161             # failed/total locales
      Utilities: 0/3               # failed/total utilities
      Branch:                      # optional branch name
      Revision: 473590             # svn revision of sources

2. Create another script running periodically, say once or a few
    times a day, as a cron job at apache.org that processes the
    emails above (e.g., by reading the stdcxx-dev mbox archive,
    http://tinyurl.com/ylo4ly), extracting the relevant data and
    adding them to the published test matrix currently at
    http://people.apache.org/~sebor/stdcxx/results/.

This way any user could run stdcxx-test on their machine and get
their test results published on the official result page. We could
also instruct the second script to only process results posted by
trusted parties to prevent pranksters from spoofing our results.

Thought? Comments?

Martin

Mime
View raw message