db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rick Hillegas <rick.hille...@oracle.com>
Subject Re: Test Methodology
Date Tue, 14 Feb 2012 14:37:07 GMT
Hi Sara,

Here are some answers to your questions.

Hope this helps,
-Rick

On 2/13/12 1:14 PM, Sara Nilsson wrote:
> To gatekeepers, testers or to whom this may concern.
>
> We're four engineering students at the Faculty of Engineering, 
> University of
> Lund (LU, http://www.lu.se ; LTH, http://www.lth.se) at the department of
> Computer Science. We're working on a paper on test methodology in open
> source projects and would be very grateful if you could spare a few 
> minutes
> and answer the following questions.
>
> Our intentions are to send this questionaire to a number of open source
> projects and compare their view of testing the project they're currently
> developing.
>
>
> When the paper is finished we're more than willing to share our findings
> with you.
>
> Sincerely
>
> Alexander Wallin, Ola Berg, Oscar Hjerpe and Sara Nilsson
> Engineering students
> Faculty of Engineering
> University of Lund
>
> Questions
>
> If you lack answers to any question please write <skip> or leave the 
> answer
> area empty.
>
> (General info about the project)
> - How large in lines of code is the project in its current form?
> (approximatively) (KLOC)
Derby has 2 repositories, one for the product source code and one for 
documentation. Here are the results of running cloc 
(http://cloc.sourceforge.net/) on these repositories:

The code repository results:

--------------------------------------------------------------------------------
Language                      files          blank        
comment           code
--------------------------------------------------------------------------------
Java                           2908         166674         
373647         634000
SQL                             307          12790          
17618          42514
XML                             130           4420           
3772          21223
HTML                             45            967            
964           8340
Bourne Again Shell               52            651           
1364           3131
CSS                               1            124             
23            709
XSLT                              4             45             
71            421
DOS Batch                        10             99              
7            291
Bourne Shell                     14             91            
250            253
Javascript                        1             75             
94            246
sed                               2             17             
59             96
XSD                               1             16             
14             67
DTD                               2             20             
43             29
--------------------------------------------------------------------------------
SUM:                           3477         185989         
397926         711320
--------------------------------------------------------------------------------

The documentation repository: This run missed the actual documentation 
source, which is maintained in DITA. The cloc tool didn't analyze the 
DITA source:

-------------------------------------------------------------------------------
Language                     files          blank        
comment           code
-------------------------------------------------------------------------------
XSLT                            11            774            
992           5416
DTD                             12            271            
517            498
XML                              3             93             
65            453
HTML                             2              4             
28             19
-------------------------------------------------------------------------------
SUM:                            28           1142           
1602           6386
-------------------------------------------------------------------------------
>
> - How many developers are currently actively developing? 
> (approximatively)
~15
>
> - For how long has this project been actively developed? 
> (approximatively)
The project was open-sourced in August, 2004, so it has been actively 
developed as an open source project for 7 1/2 years.

Before that, it was a closed-source product called Cloudscape. 
Development on Cloudscape began in 1996. So the code itself has been 
under development for 16 years.
>
> - Is there any current industry or university backers to the project?
> (Monetary and / or developers to the project)
Two large companies contribute teams to work on Derby. Those companies 
are Oracle and IBM.
>
>
> (Test related questions)
>
> - Is there any documents concerning testing?
Documentation on the Derby tests can be found here: 
http://svn.apache.org/viewvc/db/derby/code/trunk/java/testing/README.htm?view=co
> Such as a test plan, test
> strategy or equivalent documents? 

> Are they actually followed?
Yes.
>
> - Who conducts formal tests? Are there any dedicated testers? In what 
> form?
Tests are run by the active contributors. Some active contributors are 
quality engineers.
>
> - When do you decide to release a new version? Is it feature or date 
> based?
> (e.g. do you release on set dates or when new functionality is added)
Releases are built when a committer volunteers to do the leg work. The 
Derby community does not promise to produce releases on specified dates. 
However, over the last 7 years, the following release cadence has recurred:

o A feature release in the spring.

o A maintenance release in the fall.
>
> - What kind of testing is conducted before a release? Is it any different
> from when it's developed?
As a checkin barrier, committers run an extensive regression test suite. 
Other, weekly and monthly stress tests have been run as well.
>
> - Who discovers the most bugs before a release? Testers? Beta testers?
> Users?
Before the release, bugs are found mostly by contributors. Users are 
asked to test-drive patches and new features.
>
> - Who discovers the most bugs after a release? Testers? Beta testers? 
> Users?
Bugs are logged by contributors and users.
>
> - Is the majority of bugs discovered before or after a release?
We don't keep metrics on this.
>
> - Test coverage. Is the concept of test coverage important? Do you 
> have any
> statistics concerning this?
Contributors run the EMMA code coverage tool on an ad hoc basis. The 
last release for which I have published results was 10.4.1.3:

OVERALL COVERAGE SUMMARY
--------------------------------------------------
all classes: 87%  (1226/1404)
Method : 74%  (16629/22556)
Block : 69%  (486715/704201)
Line : 69%  (101022.4/146764)

OVERALL STATS SUMMARY
total packages: 90
total executable files: 1286
total classes: 1404
total methods: 22556
total executable lines: 146764

A recent (unpublished) EMMA run at Oracle produced these numbers:

all classes    93%  (1432/1546)
method    73%  (17691/24115)
block    70%  (519340/745738)
line    69%  (107361.4/156389)

>
> - What tools are used during testing?
Ant and JUnit,. In addition, Oracle and IBM drive platform tests using 
proprietary, in-house test frameworks.
>
> - Is there any difference between testing during development and during a
> release? Are developers required to test using the same tests as is used
> during a release?
During release testing:

o The regression test suite is run on a broader range of platforms than 
are stressed during development.

o Additional stress tests are run, including large-data and long-running 
tests.

o Ad-hoc integration testing of new features is performed by contributors.



Mime
View raw message