geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donald Woods <dwo...@apache.org>
Subject Re: Running integration tests by default?
Date Tue, 08 Jan 2008 15:08:15 GMT
I finally got the testsuite to run by setting up a new OpenSUSE 10.3 VM 
image with Maven 2.0.7 and Sun 1.5.0_13, but I'm still seeing the same 
mvn unit test failures when building some of the server plugins with 
car-maven-plugin.....

Guess you guys with the Macs have such a controlled OS/Java setup, that 
you don't run into the problems that we Linux and Windows users often 
have... :-)


-Donald

Donald Woods wrote:
> I have yet to get the testsuite to complete on my laptop running 
> RHEL5u1.  Guess I need to find some time to debug each failure until I 
> get down to the same number of failures Prasad gets on his automated 
> runs....
> 
> -Donald
> 
> 
> David Jencks wrote:
>>
>> On Jan 5, 2008, at 11:34 AM, Donald Woods wrote:
>>
>>> Taking another hour to run the IT tests on Tomcat and Jetty (on top 
>>> of the 30 mins. to run a full build) is just too much wasted time in 
>>> my busy schedule... :-)  Besides, we have Prasad's scheduled builds 
>>> running with IT on several times a day, which is close enough to 
>>> continuous builds to catch build breaks.
>>>
>>> I'd rather see more automated builds with every build running the IT 
>>> instead of forcing it as the default on all our developers.  If 
>>> someone is working on an isolated piece of code (like Console 
>>> portlets or non-core plugin like remote mgmt or clustering) then I 
>>> don't see how always running IT would outweigh the decreased 
>>> productivity of increased build times...
>>
>> I don't think I understand your point of view yet.  IIUC we are 
>> discussing whether we want
>>
>> A. mvn clean install runs the integration tests, and if you don't want 
>> to wait for them you can type ctrl-c at any point or run mvn clean 
>> install -P no-it
>>
>> B. mvn clean install does not run the integration tests and if you do 
>> want them you type mvn clean install -P with-it
>>
>> Obviously its not a lot of fun to wait around while the it tests run 
>> and I don't expect everyone to run them before most commits.  To me 
>> however its a good principle to run all the available tests before 
>> committing ( #1 below) so we should express that principle by making 
>> that happen in the default build ( # 2 below).  If in your judgement 
>> your changes are not likely to affect the integration tests you are 
>> free to not run them (#3 below), or indeed to not do a complete build 
>> of the server: I for one frequently don't do a full server build after 
>> changes I think are minor.  (Sometimes this results in me staying up 
>> very late trying to fix the build after I later discover I broke it)
>>
>> I understand you don't want to run the integration tests before every 
>> commit, and I don't either and don't expect to.  I'm thinking more of 
>> the case when I've worked until 3 AM and think I have everything 
>> working so I do a full build overnight as a last check before 
>> committing in the morning..... no way will I remember to type -P 
>> with-it but in this situation there's plenty of time to run all the 
>> integration tests and a big advantage to doing so.  If you are sitting 
>> around waiting for a build to complete and you forgot to type -P no-it 
>> you can still interrupt the build anytime you want -- after your 
>> module builds, after one assembly completes so you can try starting 
>> it, after some itests finish.... whenever you want.
>>
>>
>>>
>>> Also, there have been 4 console tests broken for well over a week 
>>> now, with no apparent rush to fix them, so everyone is now used to 
>>> ignoring the Build Failure notices....
>>>
>>
>> I don't think you've been following closely, or my mail isn't 
>> working.  For quite a while there was an incompatible xbean version 
>> problem with openejb which Jarek identified and fixed on Jan 2.  We 
>> then had 8 successful 2.1 builds from Jan 2 to yesterday.  We've had 3 
>> failed builds since, one possibly anomolous in geronimo-system, and 2 
>> with the broken console tests you mention.  I ran the itests before my 
>> last commit and these console tests passed, and I am not a console 
>> expert, so I haven't looked into them yet.
>>
>> However, whatever the specific facts about the current failures, I 
>> think we generally don't pay enough attention to itest failures.  
>> Personally running them myself a couple times a week is the best way 
>> I've thought of to keep them on my personal radar.  Do you have any 
>> ideas how to generally raise the level of attention paid to them?
>>
>> thanks
>> david jencks
>>
>>>
>>>
>>> -Donald
>>>
>>> David Jencks wrote:
>>>> On Jan 4, 2008, at 1:07 PM, Donald Woods wrote:
>>>>> How about not running the integration tests by default and letting 
>>>>> those who want them either 1) use a "-P testsuite" option from 
>>>>> trunk or 2) run mvn inside the testsuite subdir?
>>>> (2) is what we had before my change and didn't work for me -- I 
>>>> never ran the testsuite.
>>>> My thinking on my change is:
>>>> 1. Its better to run more tests.  For instance if you see a IT 
>>>> failure on your machine I think you are much more likely to fix it 
>>>> than if you have to wait for the notification from Prasad's 
>>>> automation.  I think this is why we have unit tests as well :-)
>>>> 2. People are lazy and don't like to type, so the default option 
>>>> should include the maximum level of testing.
>>>> 3. In the case of the testsuite, these are run after the rest of the 
>>>> build is complete, so you are free to interrupt the build if you get 
>>>> too bored/impatient.
>>>> Could you explain why you disagree with this or what factors you 
>>>> consider more important?
>>>> thanks
>>>> david jencks
>>>>>
>>>>> -Donald
>>>>>
>>>>>
>>>>> David Jencks wrote:
>>>>>> On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
>>>>>>> On Dec 31, 2007 4:41 PM, David Jencks <david_jencks@yahoo.com>

>>>>>>> wrote:
>>>>>>>> First of all I appear to have broken the build last night
with some
>>>>>>>> changes to get the roller plugin building again.  I think
I've
>>>>>>>> managed to fix all the problems -- the it tests all pass
for 
>>>>>>>> me.  Let
>>>>>>>> me know if there are still problems.
>>>>>>>>
>>>>>>>> I think its still too hard to run the integration tests.
>>>>>>>
>>>>>>> I would like to know what exactly you think is hard about it.
It 
>>>>>>> would
>>>>>>> be great if you could please share your thoughts and ideas on
making
>>>>>>> it simpler.
>>>>>> I didn't see a way to run the integration tests with the main 
>>>>>> build without a separate command.  Unless I can run everything in

>>>>>> one command, I'm pretty sure laziness will take over and I won't

>>>>>> run the integration tests.
>>>>>>>
>>>>>>>> I've made a
>>>>>>>> possibly annoying change so that the default build includes
IT.  If
>>>>>>>> you don't want them run
>>>>>>>>
>>>>>>>> mvn clean install -P no-it
>>>>>>>>
>>>>>>>> If this is too annoying we could reverse the profiles and
have the
>>>>>>>> default leave out the it as before and a with-it profile
that
>>>>>>>> includes them.
>>>>>>>
>>>>>>> Yeah. I think the default profile should not run the IT. IMHO,
I 
>>>>>>> think
>>>>>>> it should not even run the unit tests by default. Developers

>>>>>>> (should)
>>>>>>> run unit & IT tests before committing their code. And we
have
>>>>>>> automation builds with all tests that run 4 times a day anyways.
So
>>>>>>> the default profile can well do away with tests. But that may

>>>>>>> just be
>>>>>>> my opinion.
>>>>>> After working through some hard problem and getting ready to 
>>>>>> commit the last thing anyone wants to do is remember a second 
>>>>>> command or even a command line option to run some tests: they are

>>>>>> apt to type the simplest command that will check the build.  I 
>>>>>> think that command should run all the tests, including the 
>>>>>> integration tests.  Even though there's enough time to go eat 
>>>>>> dinner while they run, our most comprehensive checks will be run.
 
>>>>>> If you get too bored you can always stop the build after you think

>>>>>> enough stuff has been checked, and if you remember you can run the

>>>>>> build with options to turn off whatever tests you want to skip.
>>>>>> ApacheDS had their integration tests run using an option and most

>>>>>> people did not run them due to the extra effort of trying to 
>>>>>> remember to type more on the command line.  I think they've 
>>>>>> changed so the integration tests are run by default.
>>>>>> I'm happy to keep talking about this.... and I'll be happy enough

>>>>>> with a "with-it" option, but I think running everything by default

>>>>>> is the best strategy.
>>>>>>>
>>>>>>>>
>>>>>>>> Comments?
>>>>>>>>
>>>>>>>> This might have bad effects on Prasad's automation but I'm
not sure
>>>>>>>> how that is run.
>>>>>>>
>>>>>>> For now, the automation builds have been modified to use the

>>>>>>> no-it profile.
>>>>>>>
>>>>>> thanks!
>>>>>> david jencks
>>>>>>>>
>>>>>>>> thanks && Happy New Year!
>>>>>>>> david jencks
>>>>>>>
>>>>>>> Happy New Year to ya'll !
>>>>>>> Prasad
>>
>>

Mime
View raw message