- Build-Test Framework + Introduction

- The - - build-test framework is a simple configuration of - CruiseControl, - a framework for continuous build process. It also includes all the - necessary support to simplify installation. +

Note

+ This document is not complete. The final version will be finalized as + the framework development is completed and associated feedback is recieved. +

+ +

+ To be part of the project and successfully contribute to it this + is important to know about test and release process, automation + for testing, available test suites and scenarios. + So the main target of this document is to provide a developer with all + required information. The information is divided into the next + sections: +

+ Harmony Test and Release Process +
+ Framework for Testing and Automation +
+ Test Suites and Scenarios +

+ Harmony Test and Release Process +

+ +

+ A joint decision about a release schedule and acceptance criteria + is made on + the Harmony mailing list. + Currently only + milestone builds + are released - the project has no official releases. Usually the + milestone builds are released on 2-3 month schedule and named using + the next scheme: Apache Harmony 5.0M<milestone number>. + Before a release date there is 1 week feature freeze period and 1 + week code freeze period. During feature freeze new functionality, + big changes and code redesign are forbidden; only bugs fixes and + code tidy-up are allowed. During code freeze period only fixes for + critical/blocker issues are committed. +

+ The Harmony team does its best to deliver well tested implementation. + As the projectâs code-base consists of big number of code-lines + the testing became not a trivial task. So the clear testing process + was established, automation for testing + and several test suites were developed. +

- Currently, the framework does the following distinct tasks, in order, - and only when code changes in the repository:

Update the source for the class library from SVN
Get any new dependencies needed for building the class library
Build and test the class library
Update the source for the DRLVM from SVN
Get and build any new dependencies needed for DRLVM
Build and test the DRLVM

+ Currently the testing process consists of two testing cycles: + integrity testing and snapshot testing. + Both cycles are based on + build-test framework +

+ Integrity testing + cycle is aimed to verify that code base is buildable and runs selected + number of suites to catch regressions early. The cycle works on debug + build. The integrity testing cycle is triggered by a developer commit + (i.e. when a code is updated in the source code repository). + Results of testing are + published + almost immediately
+

+ Snapshot testing + cycle is aimed to provid tested binaries on regular basis. + Based on integrity testing results (i.e. the build is not broken and + most of tests are passing) a revision for the next snapshot is chosen. + When the snapshot is build it is uploaded to the web site and that in + turn triggers the testing cycle. For each build snapshot there + is a page with + tests results summary. + The page is updated dynamically as soon as results are ready. +

- Setup and Configuration + Framework for Testing and Automation

- + +

+ The + + build-test framework + is targeted to assist a developer with running different suites and + automation of testing process. The framework is a set of ANT scripts + that provides necessary support to simplify installation, setup of + Harmony build, test suites and configuring + CruiseControl. +

Prerequisites

You need to obtain and install the same tools, - as for building the Apache Harmony - components.

You need to obtain and install the same tools, + as for building the Apache Harmony components. +

Setup

- Get the /harmony/enhanced/buildtest/ part of the Apache - Harmony project via zip or tgz files. You also can - check it out from the project Subversion repository:

svn co http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/trunk

Get the infrastructure by checking it out from the project Subversion repository:

svn co http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/trunk/infra

- With Java, Ant and SVN installed, change into the buildtest/trunk - directory and type: -

+ With Java, Ant and SVN installed, change into the infra + directory and type: +

ant -Dtest.suites="<list of suites>" install

+ This command fetches external dependencies (including + CruiseControl), checks out Harmony build and test suites + (if necessary), sets up testing environment. After install + is complete it may request to specify unset required properties + for selected suites. The required-parameters.properties + file contains a list of all required properties. + If a property has empty or unresolved value + then a property has to be set in framework.local.properties + file and install target has to be re-run. +

+ The next step is to setup suites. (Note: the framework saves the list + of installed suites so the parameter -Dtest.suites=... + is not required any more) Just simply type: +

ant setup

- This command fetches CruiseControl, sets it up with the Apache Harmony - configuration, and checks out the software for building and testing - from Apache Harmony. -

- To run the tests successfully, you need to do the - following manual steps for now: -

- Build the classlib to create the deploy/jdk tree in classlib, - first fetching the dependencies: -
```
cd cc/projects/classlib/trunk
-ant fetch-depends
-ant
```
-
- Get the appropriate IBM J9 VM for your platform (this will go - away when DRLVM is good enough): - -
```
http://www-128.ibm.com/developerworks/java/jdk/harmony/index.html
```
-
- Unpack the above-fetched J9 distribution into the jdk directory (as - the distro assumes it is dropped into thejdk directory): - -
```
cd cc/projects/classlib/trunk/deploy/jdk/
-tar zxf Harmony-vme-linux.IA32-v4.tar.gz
```
-

+ This command launches setup target for each installed suite. +

- Starting CruiseControl + Run

- To start the system, type the following command in - buildtest/trunk: -

ant

The given command launches CruiseControl with the full test set. - To check status, point your browser to:

+ The suites' run can be launched in two ways: by the framework or by + CruiseControl tool. If a single run is needed type: +

ant run

+ The command launches run target for each installed suite. +

+ In case when a continuous build and test process is required type: +

ant run-cc

+ The given command launches CruiseControl with the full test set. + To check status, point your browser to: +

http://localhost:8080/

+ +

+ How To +

+ +

+ TODO: add instructions for "How To" setup integrity, + snapshot testing, integrate a suite/scenario +

+ +

+ Test Suites and Scenarios +

+ +

+ The Harmony testing base can be divided into 2 main parts: + suites and scenarios. +

+ A test suite is a collection of quite + simple/short test cases that is used to verify that the + implementation follows specified behavior. + The project has several own test suites as well uses 3rd party + (external) test suites. +

+ An application testing scenario (or simply testing scenario) + is a set of actions/operations that models typical application's + usage/functioning. +

Note

+ Most of suites/scenarios below can be run with and without build-test + framework. Please consult a suite's documentation for details. +

Harmony suites: +
- + Class library unit tests +
- + DRL VM unit and regression tests +
- + Functional test suite +
- + JDKTools unit tests +
- + Reliability test suite +
- + Stress test suite +
- + VM Validation test suite +
+
External test suites: +
- + Apache Ant 1.7.0 unit tests +
- + Apache Maven unit tests +
- + Eclipse unit tests. +
- + Eclipse TPTP tests +
- + Geronimo unit tests. +
- + JUnit tests +
- + Mauve test suite +
+
Application scenarios: +
- + Apache Ant scenario +
- + Apache Axis2 client-server scenario +
- + Apache Struts scenario +
- + Apache Tomcat scenario +
- + Dacapo benchmark +
- + Eclipse Geronimo application +
- + Eclipse hello world application +
- + JEdit scenario +
- + Jetty scenario +
- + LDAP scenario +
- + Scimark benchmark +
+

- The - - build-test framework is a simple configuration of - CruiseControl, - a framework for continuous build process. It also includes all the - necessary support to simplify installation. -

Note

+ This document is not complete. The final version will be finalized as + the framework development is completed and associated feedback is recieved. +

- Currently, the framework does the following distinct tasks, in order, - and only when code changes in the repository:

Update the source for the class library from SVN
Get any new dependencies needed for building the class library
Build and test the class library
Update the source for the DRLVM from SVN
Get and build any new dependencies needed for DRLVM
Build and test the DRLVM

+ Harmony Test and Release Process +
+ Framework for Testing and Automation +
+ Test Suites and Scenarios +

- - - -

You need to obtain and install the same tools, - as for building the Apache Harmony - components.

- - - -

- Get the /harmony/enhanced/buildtest/ part of the Apache - Harmony project via zip or tgz files. You also can - check it out from the project Subversion repository:

- -

svn co http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/trunk

- With Java, Ant and SVN installed, change into the buildtest/trunk - directory and type: + A joint decision about a release schedule and acceptance criteria + is made on + the Harmony mailing list. + Currently only + milestone builds + are released - the project has no official releases. Usually the + milestone builds are released on 2-3 month schedule and named using + the next scheme: Apache Harmony 5.0M<milestone number>. + Before a release date there is 1 week feature freeze period and 1 + week code freeze period. During feature freeze new functionality, + big changes and code redesign are forbidden; only bugs fixes and + code tidy-up are allowed. During code freeze period only fixes for + critical/blocker issues are committed.

- -

ant setup

- This command fetches CruiseControl, sets it up with the Apache Harmony - configuration, and checks out the software for building and testing - from Apache Harmony. + The Harmony team does its best to deliver well tested implementation. + As the projectâs code-base consists of big number of code-lines + the testing became not a trivial task. So the clear testing process + was established, automation for testing + and several test suites were developed.

- To run the tests successfully, you need to do the - following manual steps for now: + Currently the testing process consists of two testing cycles: + integrity testing and snapshot testing. + Both cycles are based on + build-test framework

- -

- Build the classlib to create the deploy/jdk tree in classlib, - first fetching the dependencies: -
```
cd cc/projects/classlib/trunk
-ant fetch-depends
-ant
```
-
- Get the appropriate IBM J9 VM for your platform (this will go - away when DRLVM is good enough): - -
```
http://www-128.ibm.com/developerworks/java/jdk/harmony/index.html
```
-
- Unpack the above-fetched J9 distribution into the jdk directory (as - the distro assumes it is dropped into thejdk directory): - -
```
cd cc/projects/classlib/trunk/deploy/jdk/
-tar zxf Harmony-vme-linux.IA32-v4.tar.gz
```
-

- - -

- To start the system, type the following command in - buildtest/trunk: + Integrity testing + cycle is aimed to verify that code base is buildable and runs selected + number of suites to catch regressions early. The cycle works on debug + build. The integrity testing cycle is triggered by a developer commit + (i.e. when a code is updated in the source code repository). + Results of testing are + published + almost immediately

ant

- -

The given command launches CruiseControl with the full test set. - To check status, point your browser to:

- -

http://localhost:8080/

- -

- - - - +

+ +

You need to obtain and install the same tools, + as for building the Apache Harmony components. +

+ + + +

Get the infrastructure by checking it out from the project Subversion repository:

+ +

svn co http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/trunk/infra

+ +

+ With Java, Ant and SVN installed, change into the infra + directory and type: +

+ +

ant -Dtest.suites="<list of suites>" install

+ +

+ The next step is to setup suites. (Note: the framework saves the list + of installed suites so the parameter -Dtest.suites=... + is not required any more) Just simply type: +

+ +

ant setup

+ +

+ This command launches setup target for each installed suite. +

+ + + +

+ The suites' run can be launched in two ways: by the framework or by + CruiseControl tool. If a single run is needed type: +

ant run

+ +

+ The command launches run target for each installed suite. +

+ +

+ In case when a continuous build and test process is required type: +

ant run-cc

+ +

+ The given command launches CruiseControl with the full test set. + To check status, point your browser to: +

http://localhost:8080/

+ + + + +

+ TODO: add instructions for "How To" setup integrity, + snapshot testing, integrate a suite/scenario +

+ +

+ The Harmony testing base can be divided into 2 main parts: + suites and scenarios. +

+ An application testing scenario (or simply testing scenario) + is a set of actions/operations that models typical application's + usage/functioning. +

Note

+ Most of suites/scenarios below can be run with and without build-test + framework. Please consult a suite's documentation for details. +

Harmony suites: +
- + Class library unit tests +
- + DRL VM unit and regression tests +
- + Functional test suite +
- + JDKTools unit tests +
- + Reliability test suite +
- + Stress test suite +
- + VM Validation test suite +
+
External test suites: +
- + Apache Ant 1.7.0 unit tests +
- + Apache Maven unit tests +
- + Eclipse unit tests. +
- + Eclipse TPTP tests +
- + Geronimo unit tests. +
- + JUnit tests +
- + Mauve test suite +
+
Application scenarios: +
- + Apache Ant scenario +
- + Apache Axis2 client-server scenario +
- + Apache Struts scenario +
- + Apache Tomcat scenario +
- + Dacapo benchmark +
- + Eclipse Geronimo application +
- + Eclipse hello world application +
- + JEdit scenario +
- + Jetty scenario +
- + LDAP scenario +
- + Scimark benchmark +
+

+ +