harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Ellison <t.p.elli...@gmail.com>
Subject [build test] documentation
Date Thu, 02 Aug 2007 11:06:33 GMT
cool - this is going straight on my reading pile!  Before I read it, may
I ask if this is appropriate material to convert to xml and move onto
the website?  It would get better visibility there IMHO.

Regards,
Tim

varlax@apache.org wrote:
> Author: varlax
> Date: Thu Aug  2 03:07:45 2007
> New Revision: 562065
> 
> URL: http://svn.apache.org/viewvc?view=rev&rev=562065
> Log:
> Additional document for HARMONY-4293
> 
> Added:
>     harmony/enhanced/buildtest/branches/2.0/dev-guide.txt   (with props)
> 
> Added: harmony/enhanced/buildtest/branches/2.0/dev-guide.txt
> URL: http://svn.apache.org/viewvc/harmony/enhanced/buildtest/branches/2.0/dev-guide.txt?view=auto&rev=562065
> ==============================================================================
> --- harmony/enhanced/buildtest/branches/2.0/dev-guide.txt (added)
> +++ harmony/enhanced/buildtest/branches/2.0/dev-guide.txt Thu Aug  2 03:07:45 2007
> @@ -0,0 +1,591 @@
> +==============================================================================
> +                              BTI Design Document
> +==============================================================================
> +:Title:   BTI Design Document - a Guide for BTI Developers
> +:Author:  Alexander Kleymenov <kleymenov{ at }gmail{ . }com>
> +:Date:    2007/07/05
> +
> +This document contains detailed description of each of the functional parts
> +of Harmony BTI. It could be used as a *Guide for BTI Developers*. 
> +For quick start with BTI please look at `README`_ file.
> +
> +Architectural parts of BTI covered in this document can be summarized
> +as follows:
> +
> +:`Execution Core`_:
> +    Maintains BTI configuration and Tests execution.
> +    Acts as a glue for other components of BTI.
> +    Provides some auxiliary functionality for implementation of 
> +    test suites adaptors.
> +
> +:`Test Suite Adaptors`_:
> +    The Ant scripts integrating the test suites to BTI by means of
> +    implementation *adapting* interface.
> +    Consist of ``adaptor.xml`` file maintaining test execution
> +    and ``parameters.xml`` file describing configuration parameters.
> +
> +:`CTE Module`_:
> +    Performs Continuous Test Execution.
> +    Is built on the base of CruiseControl system.
> +
> +:`Execution Status Notifier`_:
> +    Sends mail alerts with the status of test execution.
> +
> +:`External Dependencies Fetching Module`_:
> +    Provides the unified means for description of external dependencies
> +    used by test suites or BTI.
> +    Downloads and installs the external libraries.
> +    Allows sharing of external libraries between test suites.
> +
> +
> +Execution Core
> +==============
> +
> +Implemented by the following source files::
> +
> +    ./build.xml
> +    ./buildtest.bat
> +    ./buildtest.sh
> +    ./scripts/exec-adaptor.xml
> +    ./scripts/framework.xml
> +    ./scripts/main.xml
> +
> +This part of BTI is responsible for BTI configuration and execution.
> +It acts as an execution engine gluing all other functional parts of
> +BTI together. It also provides some functionality supporting the development
> +of Test Suite Adaptors.
> +
> +
> +BTI Configuration
> +-----------------
> +
> +The BTI configuration is made by means of the property files.
> +The first properties file::
> +
> +    ./scripts/framework.properties 
> +
> +contains the properties affecting internals of BTI and *SHOULD NOT*
> +be modified by user. The second file::
> +
> +    ./scripts/parameters.xml
> +
> +is an XML Ant properties file containing the configuration parameters for all
> +of the functional parts of BTI. These parameters can be configured by user
> +through the third configuration file::
> +
> +    ./framework.local.properties
> +
> +By means of this file user can also tune some test suites' specific parameters
> +defined in:: 
> +
> +    ./adaptors/*/parameters.xml
> +
> +files.
> +
> +The last configuration file is::
> +
> +    ./required-parameters.properties
> +
> +It is generated by BTI and contains the assignments of parameters *required* by
> +test suites selected for execution (more on this in `Test Suite Adaptors`_
> +section).
> +
> +
> +BTI Execution
> +-------------
> +
> +BTI execution is launched by means of a command line UI provided by the Core.
> +Currently this UI consists of the following Ant targets expected from user as
> +parameters to BTI command line scripts (``buildtest.bat`` for MS Windows[x]_,
> +and ``buildtest.sh`` for Linux):
> +    
> +``install``
> +    Does an installation of BTI on local workstation from remote SVN
> +    repository.
> +    Checks out (for initial execution) or updates (for consequent ones)
> +    BTI implementation files (``./scripts`` directory) and test suite
> +    adaptors (``./adaptors`` directory).
> +    This target *MUST* be run at first after initial BTI checkout (see
> +    ``./README.txt`` for how to get BTI from SVN repository).
> +    Generates ``./framework.local.properties`` property file for local BTI
> +    configuration.
> +
> +``framework-update``
> +    Does an update of current BTI installation from the SVN repository.
> +    Updates BTI implementation scripts and test suite adaptors.
> +
> +``setup``
> +    Does a setup of the test run. Expects the ``test.suites`` property 
> +    to be set to comma separated list of test suites selected for execution.
> +    Produces 3 files:
> +
> +    ``./required-parameters.properties``
> +        property file containing the assignments of parameters required 
> +        by selected test suites.
> +
> +    ``./selected-suites.properties``
> +        property file containing the  test run configuration. Describes the
> +        order in which the selected suites will be run, and dependencies
> +        between them.
> +
> +    ``./resources.location``
> +        property file describing the locations of external resources needed by
> +        configured test run.
> +
> +``run``
> +    Launches test execution. The selected test suites will be executed in
> +    order defined by ``test.suites.names.list`` property in 
> +    ``./selected-suites.properties`` file. 
> +    At first, BTI executes ``setup`` targets of adaptor scripts of all
> +    of the selected test suites (in defined order). If ``setup`` fails for
> +    some of the suite, this suite won't be ``run``, and all dependent test
> +    suites won't be ``setup`` and ``run``. BTI will warn about all of
> +    the failed and skipped test suites by means of its log messages.
> +    At second, after successful ``setup``, BTI launches ``run`` targets of
> +    selected test suites. Similarly to ``setup`` BTI will skip the execution
> +    of dependent test suites if their superior test suites failed to
> +    ``run``. The information about failed and skipped test suites will be
> +    printed out by BTI to its log.
> +    After all runs of the selected test suites BTI ends with *SUCCESS* if
> +    there were no test suites failures, and *ERROR* if some suites could not
> +    be set up or failed to run.
> +
> +``run-cc``
> +    Does the same as ``run``, but test cycle is continuously repeated. Each
> +    iteration of test suites execution is triggered whether by 
> +    modifications happened in some of the controlled by test suites SVN 
> +    repositories, or by schedule provided by user. See `CTE Module`_ for more 
> +    information on Continuous Test Execution.
> +
> +``clean``
> +    Consequently launches ``clean`` targets of adaptors of selected test 
> +    suites.
> +
> +``test-mail``
> +    Sends test mail notification. Used to check the operability of the
> +    current configurations of `Execution Status Notifier`_ module.
> +        
> +The execution of BTI can be configured by the following properties specified
> +in ``./framework.local.properties`` file::
> +
> +    framework.parameters.optional.dolog
> +    framework.parameters.repository.noupdate
> +    framework.parameters.repository.awaited
> +
> +For descriptions of these parameters look in ``./scripts/parameters.xml`` file.
> +
> +
> +
> +Test Suite Adaptors
> +===================
> +
> +A Test Suite Adaptor consists of two Ant files placed under 
> +``./adaptors/${suite.name}`` directory:
> +    
> +adaptor.xml
> +-----------
> +
> +Regular Ant project file implementing 3 targets: ``setup``, ``run``, and
> +``clean``:
> +
> +    **setup**
> +        has to perform all the works on setting of the test suite on the
> +        local workstation: it should download and build the sources for test
> +        suite, check required prerequisites, and so on - everything that
> +        should be done before first run of the test suite on the workstation.
> +        `Execution Core`_ provides some functionality which can be reused
> +        here:
> +
> +        ``<svn-co url="${url}" trunk="${trunk}"/>`` - macrodefinition checking
> +        out the SVN repository located at ``${url}`` to directory pointed by
> +        ``${trunk}``.
> +
> +        ``<svn-update-bt path="${path}"/>`` - macrodefinition updating the
> +        local ``${path}`` of BTI repository. It is useful when the sources of
> +        the test suite are located under BTI's SVN repository.
> +
> +    **run**
> +        performs the actual run of the test suite. It should "``<fail>``"
> +        if test suite's execution status is FAILED. 
> +
> +    **clean**
> +        cleans the products of test suite's execution
> +        
> +The following properties provided by `Execution Core`_ can be useful for
> +implementing these targets:
> +
> +    :``suite.name``: 
> +        holding the name of the suite
> +
> +    :``suite.adaptor.dir``: 
> +        the directory where adaptor is placed
> +
> +    :``test.sources.dir``: 
> +        the root directory containing the test sources.
> +        Usually the test sources for each particular test suite are placed
> +        under ``${test.sources.dir}/${suite.name}`` directory.
> +
> +    :``temp.files.dir``: 
> +        the directory for temporal files
> +
> +    :``checkouts.dir``: 
> +        the directory for external checkouts. For example *classlib* test
> +        suite checks out its sources to ``${checkouts.dir}/classlib`` directory.
> +
> +    :``results.dir``:
> +        the BTI's directory for storing the test execution results. Usually 
> +        test suite uses ``${results.dir}/${suite.name}`` directory for this
> +        purpose.
> +
> +    :``classes.dir``:
> +        the directory for compiled java classes. It's needed if the test suite
> +        adaptor uses some java classes which should be compiled.
> +
> +parameters.xml
> +--------------
> +
> +Ant's xml property file containing declarations of parameters describing
> +the test suite. This xml property file is loaded after
> +``adaptor.xml`` file and, thus, can contain references to the properties 
> +defined there. The parameters described in parameters.xml file can be divided 
> +into two categories: *execution* and *integration* parameters.
> +
> +Execution Parameters
> +~~~~~~~~~~~~~~~~~~~~
> +
> +are the parameters affecting the test suite execution. 
> +These parameters includes:
> +
> +*depends*
> ++++++++++
> +
> +    contains comma separated list of all of the test suites on which this test
> +    suite depends. Actually this parameter defines the order in which the
> +    test suite will be executed: it will be executed after suites defined by
> +    this parameter. If some of the listed test suites were not selected for
> +    execution, they are removed from this parameter. If default value of this
> +    parameter does not correspond to selected set of test suites, User *SHOULD*
> +    change this parameter by means of ``${suite.name}.parameters.depends``
> +    property specified in ``./framework.local.properties`` file.
> +
> +*shared* parameters 
> ++++++++++++++++++++
> +
> +    are the parameters holding the values which can be used by other test
> +    suites. Such parameters affecting the execution of other test suites.
> +    The path on built product (as JVM for example) can be an example of such
> +    parameter.
> +
> +*required* parameters 
> ++++++++++++++++++++++
> +    
> +    are the parameters for which the test suite is not able to provide the
> +    default values: the values for these parameters can be specified only
> +    from the outside of the test suite. For example: such parameters can
> +    hold the path to tested JVM - test suite can't guess this information,
> +    so it can't provide the default value for this parameter. On the BTI's
> +    ``setup`` stage, all the required parameters for the selected
> +    test suites are collected in ``./required-parameters.properties`` file and
> +    if some of such parameters has no value, BTI stops with *ERROR* message.
> +    
> +    **Note:** the declaration of *required* parameter should have a 
> +    ``description`` attribute. Without this attribute such a 
> +    parameter won't be regarded as *required*.
> +    
> +    While test suite itself can not provide the default value for such a
> +    parameter, it is possible to provide it from BTI's (read "external")
> +    side. It is done by means of connecting *required* parameter of test
> +    suite with *shared* parameter of test suite which can provide *required*
> +    value. Such test suites providing their *shared* parameters should be 
> +    listed in ``depends`` parameter of dependent test suite - it will allow
> +    them to be executed before dependent test suite, and, correspondingly, 
> +    to build a product pointed by the value of *shared* parameter before
> +    dependent test suite will use it.
> +    The values for *required* parameters can be specified in three places:
> +
> +    * test suites ``./parameters.xml`` file can contain only connections
> +      between *required* parameters and *shared* parameters of some other
> +      test suites. These test suites should be listed in ``depends`` parameter.
> +
> +      **Note:** if these test suites are not selected for execution, these
> +      *shared* parameters won't be resolved and *required* parameter won't
> +      have the value. BTI will stop with ERROR message requesting to specify
> +      the value for this unset *required* parameters.
> +
> +    * ``./framework.local.properties`` file can contain connections between
> +      the *required* and *shared* parameters, as well as direct evaluation of
> +      *required* parameters with some value. The value provided in this file
> +      takes precedence over the value in ``./parameters.xml`` file.
> +
> +    * ``./required-parameters.properties`` file can contain only direct
> +      evaluation of required parameter with some value. It can't contain any
> +      property resolutions. The value provided in this file takes precedence
> +      on the values provided in ``./framework.local.properties`` and
> +      ``./parameters.xml`` files. 
> +      
> +      **Note:** the *required* parameters in this file
> +      appears in short format: ``${suite.mane}.${required.parameter.name}``. For
> +      example, if *required* parameter has been defined in ``parameters.xml`` as
> +      ``suite.parameters.required.some.param``, in
> +      ``./required-parameters.properties`` it will be in the following form:
> +      ``suite.some.param``.
> +
> +*optional* parameters
> ++++++++++++++++++++++
> +
> +      are the parameters allowing to user to influence how the test suite
> +      will be executed. Such parameters have some default values and should
> +      have the description.
> +
> +
> +Integration Parameters
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +describe the support from BTI's side needed for correct test suite execution.
> +These parameters includes:
> +
> +*external* parameters
> ++++++++++++++++++++++
> +
> +      are the parameters describing the dependencies of the test suite on
> +      external resources located at some point of the web. This parameters are
> +      addressed to the `External Dependencies Fetching Module`_ which is
> +      responsible for downloading and sharing them between the test suites.
> +
> +*cc* parameters
> ++++++++++++++++
> +
> +      are the parameters describing test suite's specific of continuous
> +      execution. These parameters describe files and SVN repositories monitored
> +      by test suite, define bootstrapping and publishing procedures, do some
> +      test suite's specific configurations of `Execution Status Notifier`_
> +      module. For more information see `CTE Module`_ section.
> +
> +*mail* parameters
> ++++++++++++++++++
> +
> +      are the parameters describing the test suite's specific configuration of
> +      `Execution Status Notifier`_ module. Initially these parameters were
> +      *cc* parameters, but now they have been moved into special category.
> +      The parameters specified here take precedence over corresponding *cc*
> +      ones. This parameters allows to specify the summary information to be
> +      sent with mail notifications, files to be attached, and so on. See
> +      `Execution Status Notifier`_ for more details.
> +
> +
> +
> +CTE Module
> +==========
> +
> +Continuous Test Execution module consists of the following source files::
> +
> +    ./scripts/cc-project.xml
> +    ./scripts/cc-exec-adaptor.xml
> +    ./scripts/templates/cc-config-template.xml
> +    ./scripts/templates/cc-project-template.xml
> +
> +This module is responsible for performing of continuous test execution
> +launched by ``run-cc`` BTI's target. The work of this module can be divided into
> +2 steps: `preparation`_ and `execution`_.
> +
> +Preparation
> +-----------
> +
> +On this stage, CTE Module gathers the parameters describing CruiseControl-related
> +configuration and generates CC project files (CC configurations) for each of
> +selected test suites. The parameters defining these configurations are
> +provided by two sources:
> +
> +- BTI defines *global* CTE parameters common for all test suites. Such
> +  parameters include the information about web ports, timeouts, etc. and
> +  can be (along with their descriptions) seen in ``./scripts/parameters.xml``
> +  file under ``<cc>`` section.
> +
> +  **Note:**
> +  These parameters are *user configurable*. It means that their
> +  values can be changed by user through the corresponding properties
> +  specified in ``./framework.local.properties`` file.
> +
> +- Each of selected test suites uses its own set of CTE parameters
> +  describing the specific of its continuous execution. These parameters
> +  appear in test suites' ``parameters.xml`` file under ``<cc>`` section: 
> +
> +    :``trigger``: - (no value by default)
> +      - path to local file which modification will trigger the execution 
> +        of test suite.
> +
> +    :``usesvn``:  - (no value by default)
> +      - the URL of SVN repository which modification will trigger the 
> +        test suite's execution. There can several URLs be set. 
> +        Every other parameter with ``usesvn.`` prefix is considered as URL 
> +        for SVN repository.
> +
> +    :``publish``: - (default is ``true``)
> +      - determines whether do status notification publishing for this test
> +        suite or not. The value of this parameter can be changed by user
> +        through the ``./framework.local.properties`` configuration file.
> +        In future this parameter will go to ``<mail>`` section of parameters
> +        configuration file (so it won't be a CTE configuration parameter).
> +        
> +    :``results``: - (no value by default)
> +      - path to directory containing the test execution result files in XML
> +        JUnit format for status notification message. This parameter will 
> +        go to ``<mail>`` section.
> +
> +  The `Execution Status Notifier`_ configuration parameters may also
> +  appear in ``<cc>`` section. But their normal place has been moved to 
> +  ``<mail>`` section. They are: ``summary``, ``message``, and ``attachs``.
> +  They describe summary, content of the notification message and files to be
> +  attached with the message correspondingly.
> +
> +  **Note:**
> +  These parameters (except ``publish``) are `Integration Parameters`_ and
> +  are *not user configurable*.  It means that their values should not be changed
> +  by user.
> +
> +Execution
> +---------
> +
> +After ``setup`` of each of the selected test suites, BTI launches CruiseControl
> +with generated configuration. CC launches test suites in order described by
> +``test.suites.names.list`` property defined in ``./selected-suites.properties``
> +file generated on ``setup`` stage. 
> +If some of the test suites failed, dependent test suites won't be run.
> +
> +The execution log of each of the test suites
> +can be found in CC's web interface under *build artifacts* section. Web
> +interface of CC is available at localhost with
> +``framework.parameters.cc.webport`` port number which is 8080 by default.
> +
> +As stated in `README`_ file, there are two modes of continuous execution: 
> +SVN modifications triggered and scheduled runs. SVN modifications triggered runs 
> +are executed each time monitored SVN of trigger file modifications occur. 
> +Scheduled runs are launched at the particular time specified by the following
> +properties in framework.local.properties file:
> +
> +    ``framework.parameters.schedule.day=Tuesday``
> +       Do launch test suites execution on particular day. 
> +       If this property is not defined, the test run will be launched everyday.
> +
> +    ``framework.parameters.schedule.time=1530``
> +       Do launch test suites execution at this particular time.
> +       Should be defined for scheduled execution mode.
> +
> +
> +
> +Execution Status Notifier
> +=========================
> +
> +Implemented by the following source file::
> +
> +    ./scripts/publisher.xml
> +
> +This module performs the composition and sending of the message with
> +test suite execution status. The work of this module is regulated by means of
> +particular configuration parameters defined either by BTI or by
> +test suites. The parameters defined by BTI have the global influence and
> +take effect for all of the suites. Such the parameters are (look in
> +``./scripts/parameters.xml`` file for more info):
> +
> +    :``framework.parameters.usemail``:
> +          the notification publishing will be in use only if this parameter is
> +          set and contains the address of SMTP server to be used for mailing.
> +
> +    :``framework.parameters.usemail.always``:
> +          defines the notification mode. If it holds ``true`` value - the
> +          notifications will be sent regardless of execution status. 
> +          But if it holds ``false`` - notifications will be sent only after 
> +          execution status changes (from *PASSED* to *FAILED*, and vice versa)
> +
> +    :``framework.parameters.usemail.to``:
> +          should contain the address to where notifications should be sent.
> +          Use comma for multiple values.
> +
> +    :``framework.parameters.usemail.from``:
> +          should contain the address from where notifications should be sent
> +          Use comma for multiple values.
> +
> +    :``framework.parameters.usemail.suppress.exelogs``:
> +          controls whether execution logs will be sent with failure messages or
> +          not.
> +
> +    :``framework.parameters.usemail.subject.prefix``:
> +          sets up the prefix for notifications subject line.
> +
> +Also each of the test suite can provide its own parameters configuring the
> +status notifications. They are described in test suite's parameters.xml file
> +under <mail> section and their values should not be changed by user:
> +
> +    :``summary``:
> +          the path to file holding test suite's execution summary. This content
> +          will be prepended to the notification message
> +
> +    :``message``:
> +          the path to file holding test suite's message content. This content
> +          will replace one generated by default.
> +
> +    :``attachs``:
> +          comma separated paths to files to be attached with notification.
> +
> +Sometimes it is necessary for test suite to override some global parameter
> +defined by framework's parameters. It can be done by means of parameters
> +specified in ``./framework.local.properties`` file having
> +``${suite.name}.parameters.mail`` prefix instead of 
> +``framework.parameters.usemail`` one.
> +
> +For example, if the User needs for *test_suite* to use particular destination
> +mail address, the following property can do that:
> +
> +``test_suite.parameters.mail.to=different@address.info``
> +
> +In this case status notifications for all of the test suites will be sent to 
> +the address specified by ``framework.parameters.usemail.to`` parameter,
> +but notifications for *test_suite* will be sent to *different@address.info*.
> +
> +
> +
> +External Dependencies Fetching Module
> +=====================================
> +  
> +With this module BTI offers the common way to fetch, use and share external
> +libraries between the test suites. It is done by means of parameters described
> +in ``<external>`` section of ``parameters.xml`` files. The description of the

> +external library should contain at least 2 fields: URL from where to take the 
> +dependency, and MD5 value to check the dependency. Also there can be some
> +attributes describing the type of the dependency and some installation
> +information.
> +
> +BTI on ``setup`` stage fetches all the dependencies according to their
> +descriptions and sets up corresponding properties to access them.
> +For example: let's look at the following description::
> +
> +    <external>
> +        <my-dependency
> +            url="http://host.of/my-dependency.zip"
> +            md5="87A617D7E65979AE01B5ACE4649EB40B"
> +            dir="my-dep-dir"
> +            inc="*/lib/*.jar"
> +        />
> +    </external>
> +
> +It describes dependency called *my-dependency* which is located at specified
> +URL and has particular MD5 check sum. During the local installation of
> +downloaded archive only jar files from `lib` directory will be unzipped and
> +placed under `my-dep-dir` directory. The path to this directory will be
> +provided by ``ext.my-dependency.location`` property.
> +
> +This module is under development and provides only simple functionality.
> +For working usage examples, please, look at ``./scripts/parameters.xml`` file.
> +
> +------------------------------------------------------------------------------
> +
> +.. [x] Other brands and names are the property of their respective owners
> +.. _README: ./README.txt
> +
> +.. Use "Vim reStructured Text" utility to generate HTML:
> +        http://skawina.eu.org/mikolaj/vst.html
> +   or online renderer:
> +        http://www.hosting4u.cz/jbar/rest/rest.html
> +
> +.. vim:filetype=rst
> +
> 
> Propchange: harmony/enhanced/buildtest/branches/2.0/dev-guide.txt
> ------------------------------------------------------------------------------
>     svn:eol-style = native
> 
> 
> 

Mime
View raw message