harmony-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Harmony Wiki] Trivial Update of "Eclipse Unit Tests" by Vladimir Beliaev
Date Tue, 15 Apr 2008 08:36:25 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Harmony Wiki" for change notification.

The following page has been changed by Vladimir Beliaev:

New page:
[wiki:Self:DRLVMTestTracking Back to DRLVM Test Tracking]

=== Summary ===

[#Mission_Status Mission and Status][[BR]]
[#How_to_run_EUT How to run EUT]
 1. [#prepare_env             Prepare Common Environment]
 2. [#How_to_run_EUT_BTI      Running EUT within Build & Tests infrastructure]
 3. [#How_to_run_EUT_runtests Running EUT using its own scripts]
 4. [#How_to_run_EUT_testcase Running EUT testcase with java command]
 5. [#How_to_run_EUT_GUI      Running EUT under Eclipse GUI]
[#EUT33_information_table EUT3.3 information table]

=== Mission and Status ===

Test Harmony with Eclipse Unit Tests (version 3.3) to achieve 98% pass rate for all available

EUT is currently demonstrates over 98% pass rate on both Windows (2003 Server) and Linux (SLES10)
x86 platforms. Please see the testing results at automated testing report page which is:


[#Summary Back to Summary]

=== How to run EUT ===

=== 1. Prepare Common Environment ===

'''''1.1 Download Eclipse SDK and Eclipse Unit Tests 3.3 archives from:'''''


where the tests archive name is:


and depending on platform the Eclipse SDK archive names are: 


'''''1.2 Choose Reference Java to launch Eclipse'''''

EUT architecture allows to use two runtimes - one to launch Eclipse and parse EUT setup files
and second one is to run tests themselves. Using Harmony as both runtimes is the subject of
future work. For now `HotSpot VM` should be used as the Eclipse launcher - just add this java
to path like:

export PATH=<path to HostSpot Server VM 1.5.0>

'''''1.3. Patch SWT suite or set the stable proxy'''''

If you are running EUT from outside the firewall or do not run `swt` suite then just skipt
this step. 

Recently it was discovered (HARMONY-5306) that `swt` suite accesses web pages and it hangs
if such a page is unavailable (regardless to the runtime being used). So one needs to setup
a stable proxy (it looks like that `swt` takes it from Mozilla or IExplorer settings) or rebuild
`swt` test suite classes and data to replace external URLs to any internal one which does
not require a proxy to be accessed. The patching is described in [#How_to_run_EUT_BTI 2. Running
EUT within Build & Tests infrastructure] section below.

'''''1.4. Prepare Windows environment'''''

1.4.1 On Windows the EUT must be run from `Command Prompt` (running from `Cygwin` is not tested).

1.4.2 EUT requires `unzip` command to be available, its documentation recommends using ''Info-Zip''
one (`http://www.info-zip.org`).

1.4.3 There is a path length restriction on Windows for EUT, so the safest way to do not break
it is to choose the working directory as close to root as possible (like `c:\tmp-eut33`).

'''''1.5 Prepare Linux environment'''''

1.5.1 EUT is a GUI application. You need to care about correctly configuration and starting
X-server/connection if you run EUT on remote host.

1.5.2 As a temporary workaround for HARMONY-2914 set open files limit to 16256 as:
$ulimit -n 16256
$ulimit -n
You may need to have the root rights to increase the hard limit before setting the soft one
as written above.

'''Note:''' the remained steps of environment preparation depend on a way of EUT runnning
which are described below.

==== 2. Running EUT within Build & Tests infrastructure ====

There is a Cruise Control based automated testing infrastructure. In particular it contains
the ant-based script to run EUT. It is the safest way to run EUT because this script does
most of preparation work - you need just to download the archives, provide the path to Harmony
JDK and run the script.

'''''2.1 Checkout Build & Tests infrastructure (BTI):'''''
svn checkout -r HEAD https://svn.apache.org/repos/asf/harmony/enhanced/buildtest/branches/2.0/tests/eut

Enter the resulted `eut33` directory as the main work directory for the steps below.

To proceed using the checked-out scripts you need to add ant1.6.5 (or later) to your environment
export ANT_HOME=<path to ant 1.6.5 or later>
export PATH=$ANT_HOME/bin:$PATH

You may also need to care about proxy settings for ANT if you work from inside the firewall:

export ANT_OPTS=-Dhttp.proxyHost=<your proxy host> -Dhttp.proxyPort=<your proxy port>

'''''2.2 Place the downloaded archives into checkout directory''''' like (for Linux x86_64):
cp <eclipse-Automated-Tests-3.3.zip> .
mkdir -p eclipse-testing
cp <eclipse-SDK-3.3-linux-gtk-x86_64.tar.gz> eclipse-testing/.

If you do not have these archives downloaded yet then next step with gets them.

'''''2.3 Build utilities classes'''''

ant setup

This builds a set of utility classes required for EUT results processing. Also this downloads
and places accordingly the EUT archives if they are not available yet - like described in
#1.1 and #2.2 above.

'''''2.4 Patch EUT3.3 data'''''

For now one needs to patch the `swt` suite data if one runs EUT from inside the unstable firewall
(see HARMONY-5306 for details). Pick the local site you want to access from EUT (say, `http://localhost:8080`)
and execute the steps below:

cd extra/eut.3.3.swt.patching

Copy already downloaded archives to avoid one more downloading from web:
cp ../../eclipse-Automated-Tests-3.3.zip .
cp ../../eclipse-testing/eclipse-SDK-3.3-win32.zip .

Edit 'properties' file to specify your local URL:
cat properties | sed -e 's@^swt.*@swt.local.url=http://localhost:8080@' > X
mv X properties

Do the tests patching by simply executing:

Save the patched archive for future use, copy it to `eut33` root directory at least:
cp ./patched/eclipse-Automated-Tests-3.3.zip ../../.

'''''2.5 Run EUT with BTI scripts'''''

Now you are ready to run EUT. For example to run a single suite - jdtdebug - execute the following
command from eut33 directory:

ant -Dtest.jre.home=<path to harmony jdk> -Dvm.options="-Xmx1024m" -Dtests=jdtdebug

You can run several suites like:

ant -Dtest.jre.home=<path to harmony jdk> -Dvm.options="-Xmx1024m" -Dtests="jdtdebug,compare,ant"

Or all available suites by simply omitting "tests" setting:

ant -Dtest.jre.home=<path to harmony jdk> -Dvm.options="-Xmx1024m"

You may define these properties also in `eut.properties` file.

'''Note:''' there is no need to clean-up EUT files between runs - the latest version of EUT/BTI
cares about such a cleaning before each suite run.

'''''2.6 Understanding EUT-from-BTI results'''''

If there is no internal errors in BTI scripts then the `results/latest/index.html` files is
created which contains the summary of the results of recent run. This file contains the following

 * relative summary - the relative pass rate for the suites you have just run, so it can be
100% if you run a single suite and all its tests passed
 * absolute summary - it shows the absolute pass rate calculated based on total tests number
in EUT which is over 40'000
 * link to output.txt - contains EUT run output (both standard and error)
 * link to report.txt - contains unexpected failures/errors details (like stack traces) collected
from correspondent xml reports, recently the number of failures/errors details was limited
to reduce report.txt size.
 * link to eut.efl file - there is no eXclude list for EUT, all of the tests are run, still
the results are filtered based on Expected Failures List (EFL)
 * Suites Details information - the following rules are used to create this table:
  * if the suite was not run then it is gray in the table
  * if the suite ended normally (without crash or hang) and related html report was generated
then the suite record is linked to this html file
  * if the suite crashed or hanged (and killed by timeout) then it is highlighted with red
  * if the suite contains no unexpected errors/failures then no errors/failures are given
for it in this summary table

[#Summary Back to Summary]

==== 3. Running EUT using its own scripts ====

Suppose that you've successfully done all steps from the section #1 above, and EUT related
archives are placed in the current directory... Then:

'''''3.1 Unpack tests and move SDK to 'eclipse-testing':'''''

unzip -qq eclipse-Automated-Tests-3.3.zip
cd eclipse-testing    
mv ../eclipse-SDK-3.3-linux-gtk-x86_64.tar.gz .

You entered the resulted `eclipse-testing` directory which is the main work directory for
the steps below.

'''''3.2 Prepare properties file:'''''

Create any property file (say, `eut.run.properties`) with the content like:

J2SE-5.0=<your Harmony JDK Home>/bin/java
jvm=<your Harmony JDK Home>/bin/java
extraVMargs=-showversion -Xmx1024M -Duser.home=<your home> -Djava.io.tmpdir=<your
user.home=<your home>
java.io.tmpdir=<your temp>

You may miss the `user.home` and `java.io.tmpdir` redefinition still it is recommended to
do this settings as well as removing all data from these directories before each run of suite.

If you are running `teamcvs` suite you need to configure cvs server (this subject is out of
this document scope) and add the cvs-related settings to properties file like:

cvs_user=<account name used to connect to the cvs repository>
cvs_password=<the account password>
cvs_host=<the name of cvs server>
cvs_root=<the repository path>

'''''3.3 Patch `runtests` script''''' (Linux only):

You need to comment out the DISPLAY setting in the `runtests` script. For example:

cat runtests | sed -e 's@^DISPLAY@#DISPLAY@' > X
mv X runtests
chmod a+x runtests

'''''3.4 Run EUT with its own scripts'''''

Now you are ready to run EUT. To run all of the suites on SLES10 x86_64 execute the command:

./runtests -properties eut.run.properties -os linux -ws gtk -arch x86_64

Execute the same command appended by particular suite name to run this suite only like:

./runtests -properties eut.run.properties -os linux -ws gtk -arch x86_64 jdtdebug

You need to clean all of data created by Eclipse/EUT to insure the accurate run of next suite.
The safest way to do this is to repeat all steps starting from 3.1 above before each suite

'''''3.4 Understanding EUT-run-by-own-scripts results'''''

The results/logs files are located at `results` directory. For example, to get the result
of `jdtdebug` suite run first of all see the file:


[#Summary Back to Summary]

==== 4. Running EUT testcase with java command ====

Suppose that you've successfully done all steps from the section #1 above, and EUT related
archives are placed in the current directory... Then:

'''''4.1 Unpack tests and SDK:'''''

For example, for SLES10 x86_64 execute the following:

unzip -qq eclipse-Automated-Tests-3.3.zip
cd eclipse-testing
unzip -qq eclipse-junit-tests-I20070625-1500.zip
tar xzf ../eclipse-SDK-3.3-linux-gtk-x86_64.tar.gz

In case of `eclipse-SDK-3.3-win32.zip` unzipping you need to allow overiding to `All` files.

You entered the resulted `eclipse-testing` directory which is the main work directory for
the steps below.

'''''4.2 Detect command line parameters for suite'''''

First, get the failed suite class name, for example from main page of EUT report on automated
testing page (see URL above). An example of such a name is `org.eclipse.jdt.debug.tests.AutomatedSuite`.

Then you can get the plugin name based on package name of the class you found from `eclipse-testing/test.xml`
from the structure like:
<target name="jdtdebug">
        <runTests testPlugin="org.eclipse.jdt.debug.tests" />

Note, that sometimes one plugin keeps several suites, so there may be no 1-to-1 correspondance
- be creative to derive the plugin name.

Finally you need to detect if the suite is `uitestapplication` or `coretestapplication`. Inspect
the `test.xml` of related plugin to get this like:


and look for `ant target=` pattern in it like:

    <ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}">

So, if the target is `ui-test` then you get `uitestapplication`, otherwise it is `coretestapplication`.

'''Another''' way to get this data is to just follow the "Properties>>" link at the
right bottom part of suite report (in case such a report is available) like:

and inspect the `eclipse.commands` value (to long to be listed here). For example for `jdtdebug`
suite you may see:

-application org.eclipse.test.uitestapplication 
-testPluginName org.eclipse.jdt.debug.tests 
-className org.eclipse.jdt.debug.tests.AutomatedSuite

'''The simplest''' way is to use the table below in the [#EUT33_information_table EUT3.3 information
table] section.

'''''4.3 Run EUT with java command'''''

You are ready to run EUT suite with java command. For example, for jdtdebug suite on Linux
x86_64 the command looks like:

export PATH=$JAVA_HOME/bin:$PATH 

${JAVA_HOME}/bin/java \ 
        -showversion \ 
        -jar eclipse/plugins/org.eclipse.equinox.launcher_1.0.0.v20070606.jar \ 
        -application org.eclipse.test.uitestapplication \ 
        -dev bin \ 
        -data workspace \ 
        -testpluginname org.eclipse.jdt.debug.tests \ 
        -classname org.eclipse.jdt.debug.tests.AutomatedSuite \ 
        2>&1 | tee log.txt

If you need to specify more properties to this run you need to use the standard way to pass
them like:

       -Duser.home=<your home> \
       -Djava.io.tmpdir=<your temp> \
       -jar eclipse/plugins/org.eclipse.equinox.launcher_1.0.0.v20070606.jar \

'''''4.4 Understanding EUT-run-with-java-command results'''''

The EUT/suite results are located at `log.xml` (because this `log.xml` was specified for `formatter`
value in the command above). For example, one of the first lines gives the status of suite
run like:

<testsuite errors="0" failures="3" name="AutomatedSuite" package="org.eclipse.jdt.debug.tests"
tests="598" time="517.00">

Also the process standard/error output was redirected to `log.txt` - you may want to see it

==== 5. Running EUT under Eclipse GUI ====

Finally you may run EUT as JUnit Plug-in Tests in Eclipse project (which is very helpful for
single test debugging). 

You need to unpack tests and SDK as described in 4.1 section above. After that you need to
do the following steps:

 * Run `eclipse` from `<path to your eclipse-testing>/eclipse` directory
 * Select new workspace directory - make sure you control this location to clean up it before
next run of Eclipse
 * Select menu `"File"->"Import"`
 * In the newly appeared `"Import"` dialog expand `"Plug-in Development"` and select `"Plug-ins
and Fragments"`, click  `"Next"` button
 * In the `"Plug-ins and Fragments"` dialog:
  * Uncheck the checkbox `"The target platform (as specified in the Preferences)"` and make
sure that `"Plug-in Location"` is set to `<path to your eclipse-testing>/eclipse`
  * Click `"Source Code Locations..."` button - in the opened `"Preferences"` dialog go to
`"Source code locations"` tab, click `"Add..."`  button and select the following location:

<path to your eclipse-testing>/eclipse/plugins/org.eclipse.sdk.tests.source_3.2.0.v20070607/src
  * Click `"Ok"` to submit your selection, then click `"Ok"` to close `"Preferences"` dialog
  * In the `"Import As"` pane select `"Projects with source folders"` radio button and click
 * In `"Selection"` dialog select the test plugin name from the list (like `org.eclipse.jdt.debug.tests`),
click `"Add -->"` button and `"Finish"` button.

Now the test suite is imported as an Eclipse project and should be built (check `"Build Automatically"`
in the `"Project"` menu if unchecked). When it shows no errors, do the following: 
 * Select the project in `"Package Explorer"` view (`Alt+Shift+Q`,`P`)
 * Go to menu `"Run"->"Open Run Dialog..."`
 * Select `"JUnit Plug-in Test"` here and click a `"New"` icon in the left top corner of the
dialog window
 * In the right pane of this `"Run"` dialog: 
  * Choose the `"Test"` tab, check `"Run a single test"` and specify here:
   * plugin name like `org.eclipse.jdt.debug.tests` as `"Project:"`
   * test class name like `org.eclipse.jdt.debug.tests.AutomatedSuite` as `"Test class:"`
  * Then go to the "Main" tab, select `"Run an application:"` radio button and choose `"org.eclipse.ui.ide.workbench"`
from the drop-down-list. 
  * In the same tab click to `"Installed JREs"` button and specify the path to Harmony JRE,
then select it as "Runtime JRE" 
  * Finally click the `"Run"` button.

In a minute the suite will be launched and you will have "Junit" view opened with test and
testcases tree. You can use this view to re-run one test class (or testcase) if  necessary.

[#Summary Back to Summary]

==== EUT3.3 information table ====

||<tablestyle="white-space: nowrap; text-align: left;" rowbgcolor="#c0c0c0">'''Suite
name'''||'''Plugin name'''||'''Application type'''||'''Test Class'''||'''Tests'''||'''Platform'''||
||~-uiworkbenchtexteditor-~||~-org.eclipse.ui.workbench.texteditor.tests-~||ui||~- org.eclipse.ui.workbench.texteditor.tests.WorkbenchTextEditorTestSuite-~||<)>56||<)>all||

[#Summary Back to Summary]

[wiki:Self:DRLVMTestTracking Back to DRLVM Test Tracking]

View raw message