Mailing-List: contact dev-help@geronimo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@geronimo.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <477FDBB8.6030704@apache.org>
Date: Sat, 05 Jan 2008 14:34:16 -0500
From: Donald Woods <dwoods@apache.org>
User-Agent: Thunderbird 2.0.0.6 (X11/20070812)
MIME-Version: 1.0
To: dev@geronimo.apache.org
Subject: Re: Running integration tests by default?
References: <2553C225-E8ED-46EA-899B-0FCA8852FF22@yahoo.com>
 <c278ac970801020758q2d7ba42bv669c6a550e3223d6@mail.gmail.com>
 <412D2B81-6403-4EA9-8EBE-6AE0138C371E@yahoo.com>
 <477EA015.5010000@apache.org>
 <22C702C6-B36B-4EAA-AFE2-859AAE6034CF@yahoo.com>
In-Reply-To: <22C702C6-B36B-4EAA-AFE2-859AAE6034CF@yahoo.com>
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
 micalg=sha1; boundary="------------ms040901080604060909040406"

This is a cryptographically signed message in MIME format.

--------------ms040901080604060909040406
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Taking another hour to run the IT tests on Tomcat and Jetty (on top of 
the 30 mins. to run a full build) is just too much wasted time in my 
busy schedule... :-)  Besides, we have Prasad's scheduled builds running 
with IT on several times a day, which is close enough to continuous 
builds to catch build breaks.

I'd rather see more automated builds with every build running the IT 
instead of forcing it as the default on all our developers.  If someone 
is working on an isolated piece of code (like Console portlets or 
non-core plugin like remote mgmt or clustering) then I don't see how 
always running IT would outweigh the decreased productivity of increased 
build times...

Also, there have been 4 console tests broken for well over a week now, 
with no apparent rush to fix them, so everyone is now used to ignoring 
the Build Failure notices....


-Donald

David Jencks wrote:
> 
> On Jan 4, 2008, at 1:07 PM, Donald Woods wrote:
> 
>> How about not running the integration tests by default and letting 
>> those who want them either 1) use a "-P testsuite" option from trunk 
>> or 2) run mvn inside the testsuite subdir?
> (2) is what we had before my change and didn't work for me -- I never 
> ran the testsuite.
> 
> My thinking on my change is:
> 
> 1. Its better to run more tests.  For instance if you see a IT failure 
> on your machine I think you are much more likely to fix it than if you 
> have to wait for the notification from Prasad's automation.  I think 
> this is why we have unit tests as well :-)
> 2. People are lazy and don't like to type, so the default option should 
> include the maximum level of testing.
> 3. In the case of the testsuite, these are run after the rest of the 
> build is complete, so you are free to interrupt the build if you get too 
> bored/impatient.
> 
> Could you explain why you disagree with this or what factors you 
> consider more important?
> 
> thanks
> david jencks
> 
>>
>> -Donald
>>
>>
>> David Jencks wrote:
>>> On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
>>>> On Dec 31, 2007 4:41 PM, David Jencks <david_jencks@yahoo.com> wrote:
>>>>> First of all I appear to have broken the build last night with some
>>>>> changes to get the roller plugin building again.  I think I've
>>>>> managed to fix all the problems -- the it tests all pass for me.  Let
>>>>> me know if there are still problems.
>>>>>
>>>>> I think its still too hard to run the integration tests.
>>>>
>>>> I would like to know what exactly you think is hard about it. It would
>>>> be great if you could please share your thoughts and ideas on making
>>>> it simpler.
>>> I didn't see a way to run the integration tests with the main build 
>>> without a separate command.  Unless I can run everything in one 
>>> command, I'm pretty sure laziness will take over and I won't run the 
>>> integration tests.
>>>>
>>>>> I've made a
>>>>> possibly annoying change so that the default build includes IT.  If
>>>>> you don't want them run
>>>>>
>>>>> mvn clean install -P no-it
>>>>>
>>>>> If this is too annoying we could reverse the profiles and have the
>>>>> default leave out the it as before and a with-it profile that
>>>>> includes them.
>>>>
>>>> Yeah. I think the default profile should not run the IT. IMHO, I think
>>>> it should not even run the unit tests by default. Developers (should)
>>>> run unit & IT tests before committing their code. And we have
>>>> automation builds with all tests that run 4 times a day anyways. So
>>>> the default profile can well do away with tests. But that may just be
>>>> my opinion.
>>> After working through some hard problem and getting ready to commit 
>>> the last thing anyone wants to do is remember a second command or 
>>> even a command line option to run some tests: they are apt to type 
>>> the simplest command that will check the build.  I think that command 
>>> should run all the tests, including the integration tests.  Even 
>>> though there's enough time to go eat dinner while they run, our most 
>>> comprehensive checks will be run.  If you get too bored you can 
>>> always stop the build after you think enough stuff has been checked, 
>>> and if you remember you can run the build with options to turn off 
>>> whatever tests you want to skip.
>>> ApacheDS had their integration tests run using an option and most 
>>> people did not run them due to the extra effort of trying to remember 
>>> to type more on the command line.  I think they've changed so the 
>>> integration tests are run by default.
>>> I'm happy to keep talking about this.... and I'll be happy enough 
>>> with a "with-it" option, but I think running everything by default is 
>>> the best strategy.
>>>>
>>>>>
>>>>> Comments?
>>>>>
>>>>> This might have bad effects on Prasad's automation but I'm not sure
>>>>> how that is run.
>>>>
>>>> For now, the automation builds have been modified to use the no-it 
>>>> profile.
>>>>
>>> thanks!
>>> david jencks
>>>>>
>>>>> thanks && Happy New Year!
>>>>> david jencks
>>>>
>>>> Happy New Year to ya'll !
>>>> Prasad
> 
> 

--------------ms040901080604060909040406
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJAzCC
AtwwggJFoAMCAQICEGl0fFJRFoUogDNHr2j130gwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE
BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT
I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MDQwMzE5MDgwOFoX
DTA4MDQwMjE5MDgwOFowQzEfMB0GA1UEAxMWVGhhd3RlIEZyZWVtYWlsIE1lbWJlcjEgMB4G
CSqGSIb3DQEJARYRZHdvb2RzQGFwYWNoZS5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQDENBaTcon2Q3/v2plog9ympNR5IWM94JDvXhWanc1Qz07/sIerwI99WZ4z0AAV
wBelazRFCsXkFCaNDM1sOJ0gx4I+KISb5XoX/omAhYIs+Cwac2hitDraZE2Q7x281nLn4TPR
d29GVpUZVKsaDvmvnDAagGTEkOfHBx/yUihzshJn4S6n6vahimEWWXn5Qw4Oj5Muvz8iKGoA
MilrHZrfSKvCqT0LhGaFnZhrMvqeCguf0/IlP6YJ2kBmouV3XHFUaiai4PQZ7ahtbW7meKb4
b25QUjIFJHIr0KhJXqoGADRAlBgTaqWBIKbdTmYBsr55aasjp5TU+XAxiqFr1FoTAgMBAAGj
LjAsMBwGA1UdEQQVMBOBEWR3b29kc0BhcGFjaGUub3JnMAwGA1UdEwEB/wQCMAAwDQYJKoZI
hvcNAQEFBQADgYEAJReGzfwY5+BY6PuXJQLURKni6kBaFi7Z7b+tjn6iTdlZhbSF9YHazHIJ
9J6oaOzfG28JroaAlk9PeAvbC38f2kaU4bWQgm6yp6308ZRyM8V3CHXe6Wm9bXiJeZCRTpix
Mvxo+mWG2Jq0vnfjWxt+Fbi0ic1xl3I+1UguMfAfAggwggLcMIICRaADAgECAhBpdHxSURaF
KIAzR69o9d9IMA0GCSqGSIb3DQEBBQUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3
dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJl
ZW1haWwgSXNzdWluZyBDQTAeFw0wNzA0MDMxOTA4MDhaFw0wODA0MDIxOTA4MDhaMEMxHzAd
BgNVBAMTFlRoYXd0ZSBGcmVlbWFpbCBNZW1iZXIxIDAeBgkqhkiG9w0BCQEWEWR3b29kc0Bh
cGFjaGUub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxDQWk3KJ9kN/79qZ
aIPcpqTUeSFjPeCQ714Vmp3NUM9O/7CHq8CPfVmeM9AAFcAXpWs0RQrF5BQmjQzNbDidIMeC
PiiEm+V6F/6JgIWCLPgsGnNoYrQ62mRNkO8dvNZy5+Ez0XdvRlaVGVSrGg75r5wwGoBkxJDn
xwcf8lIoc7ISZ+Eup+r2oYphFll5+UMODo+TLr8/IihqADIpax2a30irwqk9C4RmhZ2YazL6
ngoLn9PyJT+mCdpAZqLld1xxVGomouD0Ge2obW1u5nim+G9uUFIyBSRyK9CoSV6qBgA0QJQY
E2qlgSCm3U5mAbK+eWmrI6eU1PlwMYqha9RaEwIDAQABoy4wLDAcBgNVHREEFTATgRFkd29v
ZHNAYXBhY2hlLm9yZzAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBQUAA4GBACUXhs38GOfg
WOj7lyUC1ESp4upAWhYu2e2/rY5+ok3ZWYW0hfWB2sxyCfSeqGjs3xtvCa6GgJZPT3gL2wt/
H9pGlOG1kIJusqet9PGUcjPFdwh13ulpvW14iXmQkU6YsTL8aPplhtiatL5341sbfhW4tInN
cZdyPtVILjHwHwIIMIIDPzCCAqigAwIBAgIBDTANBgkqhkiG9w0BAQUFADCB0TELMAkGA1UE
BhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMRowGAYD
VQQKExFUaGF3dGUgQ29uc3VsdGluZzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNl
cyBEaXZpc2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIENBMSswKQYJ
KoZIhvcNAQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAw
MFoXDTEzMDcxNjIzNTk1OVowYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25z
dWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJ
c3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1
BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAKMNcCY1osiRVwjt3J8CuFWqo/cVbLrzwL
B+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTXp6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZ
cmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYBAf8CAQAwQwYDVR0fBDwwOjA4oDagNIYy
aHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBlcnNvbmFsRnJlZW1haWxDQS5jcmwwCwYD
VR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRlTGFiZWwyLTEzODAN
BgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2asZw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82
L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSeJVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr3
94fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHTHUb/XV9lTzGCA2QwggNgAgEBMHYwYjEL
MAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAq
BgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAhBpdHxSURaFKIAz
R69o9d9IMAkGBSsOAwIaBQCgggHDMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZI
hvcNAQkFMQ8XDTA4MDEwNTE5MzQxNlowIwYJKoZIhvcNAQkEMRYEFD2eLZbe6xE25X0/Jk+k
JP/yg/o9MFIGCSqGSIb3DQEJDzFFMEMwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0G
CCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGFBgkrBgEEAYI3EAQxeDB2
MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQu
MSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQaXR8UlEW
hSiAM0evaPXfSDCBhwYLKoZIhvcNAQkQAgsxeKB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQK
ExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29u
YWwgRnJlZW1haWwgSXNzdWluZyBDQQIQaXR8UlEWhSiAM0evaPXfSDANBgkqhkiG9w0BAQEF
AASCAQBnT6Jifqf0k1k9KEKtbqg2eCsBEPYgkBTqB75kGgUERiM4n4IpzG9TWcKaJOETr6/b
WsE62rrGe62hX1S5UVbubIZOrhksMVASsZtAcxjfSA5RaDUZwxhP85ZzhE1+PfanqOvtdnv5
s0E4Y8VByZfs7iNYbuIYBeTA1Vdz1+8hLEe++WR48GaONzxtBCfMyx9+KwtlZHjjy6a7k1gL
xOyqAhajumG2RMVALDapoWF8jicqdiiLhXfQtssVFmp7YWsfGfumYTX+xP8/C+tnVCBnmkhW
HJH5W+918KZJVg7drWACBIS7gZpT11RtUumSSW72DXdyjANPICj16IaJqQKzAAAAAAAA
--------------ms040901080604060909040406--