geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donald Woods <dwo...@apache.org>
Subject Re: Running integration tests by default?
Date Mon, 07 Jan 2008 02:20:41 GMT
I have yet to get the testsuite to complete on my laptop running 
RHEL5u1.  Guess I need to find some time to debug each failure until I 
get down to the same number of failures Prasad gets on his automated 
runs....

-Donald


David Jencks wrote:
> 
> On Jan 5, 2008, at 11:34 AM, Donald Woods wrote:
> 
>> Taking another hour to run the IT tests on Tomcat and Jetty (on top of 
>> the 30 mins. to run a full build) is just too much wasted time in my 
>> busy schedule... :-)  Besides, we have Prasad's scheduled builds 
>> running with IT on several times a day, which is close enough to 
>> continuous builds to catch build breaks.
>>
>> I'd rather see more automated builds with every build running the IT 
>> instead of forcing it as the default on all our developers.  If 
>> someone is working on an isolated piece of code (like Console portlets 
>> or non-core plugin like remote mgmt or clustering) then I don't see 
>> how always running IT would outweigh the decreased productivity of 
>> increased build times...
> 
> I don't think I understand your point of view yet.  IIUC we are 
> discussing whether we want
> 
> A. mvn clean install runs the integration tests, and if you don't want 
> to wait for them you can type ctrl-c at any point or run mvn clean 
> install -P no-it
> 
> B. mvn clean install does not run the integration tests and if you do 
> want them you type mvn clean install -P with-it
> 
> Obviously its not a lot of fun to wait around while the it tests run and 
> I don't expect everyone to run them before most commits.  To me however 
> its a good principle to run all the available tests before committing ( 
> #1 below) so we should express that principle by making that happen in 
> the default build ( # 2 below).  If in your judgement your changes are 
> not likely to affect the integration tests you are free to not run them 
> (#3 below), or indeed to not do a complete build of the server: I for 
> one frequently don't do a full server build after changes I think are 
> minor.  (Sometimes this results in me staying up very late trying to fix 
> the build after I later discover I broke it)
> 
> I understand you don't want to run the integration tests before every 
> commit, and I don't either and don't expect to.  I'm thinking more of 
> the case when I've worked until 3 AM and think I have everything working 
> so I do a full build overnight as a last check before committing in the 
> morning..... no way will I remember to type -P with-it but in this 
> situation there's plenty of time to run all the integration tests and a 
> big advantage to doing so.  If you are sitting around waiting for a 
> build to complete and you forgot to type -P no-it you can still 
> interrupt the build anytime you want -- after your module builds, after 
> one assembly completes so you can try starting it, after some itests 
> finish.... whenever you want.
> 
> 
>>
>> Also, there have been 4 console tests broken for well over a week now, 
>> with no apparent rush to fix them, so everyone is now used to ignoring 
>> the Build Failure notices....
>>
> 
> I don't think you've been following closely, or my mail isn't working.  
> For quite a while there was an incompatible xbean version problem with 
> openejb which Jarek identified and fixed on Jan 2.  We then had 8 
> successful 2.1 builds from Jan 2 to yesterday.  We've had 3 failed 
> builds since, one possibly anomolous in geronimo-system, and 2 with the 
> broken console tests you mention.  I ran the itests before my last 
> commit and these console tests passed, and I am not a console expert, so 
> I haven't looked into them yet.
> 
> However, whatever the specific facts about the current failures, I think 
> we generally don't pay enough attention to itest failures.  Personally 
> running them myself a couple times a week is the best way I've thought 
> of to keep them on my personal radar.  Do you have any ideas how to 
> generally raise the level of attention paid to them?
> 
> thanks
> david jencks
> 
>>
>>
>> -Donald
>>
>> David Jencks wrote:
>>> On Jan 4, 2008, at 1:07 PM, Donald Woods wrote:
>>>> How about not running the integration tests by default and letting 
>>>> those who want them either 1) use a "-P testsuite" option from trunk 
>>>> or 2) run mvn inside the testsuite subdir?
>>> (2) is what we had before my change and didn't work for me -- I never 
>>> ran the testsuite.
>>> My thinking on my change is:
>>> 1. Its better to run more tests.  For instance if you see a IT 
>>> failure on your machine I think you are much more likely to fix it 
>>> than if you have to wait for the notification from Prasad's 
>>> automation.  I think this is why we have unit tests as well :-)
>>> 2. People are lazy and don't like to type, so the default option 
>>> should include the maximum level of testing.
>>> 3. In the case of the testsuite, these are run after the rest of the 
>>> build is complete, so you are free to interrupt the build if you get 
>>> too bored/impatient.
>>> Could you explain why you disagree with this or what factors you 
>>> consider more important?
>>> thanks
>>> david jencks
>>>>
>>>> -Donald
>>>>
>>>>
>>>> David Jencks wrote:
>>>>> On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
>>>>>> On Dec 31, 2007 4:41 PM, David Jencks <david_jencks@yahoo.com>
wrote:
>>>>>>> First of all I appear to have broken the build last night with
some
>>>>>>> changes to get the roller plugin building again.  I think I've
>>>>>>> managed to fix all the problems -- the it tests all pass for
me.  
>>>>>>> Let
>>>>>>> me know if there are still problems.
>>>>>>>
>>>>>>> I think its still too hard to run the integration tests.
>>>>>>
>>>>>> I would like to know what exactly you think is hard about it. It

>>>>>> would
>>>>>> be great if you could please share your thoughts and ideas on making
>>>>>> it simpler.
>>>>> I didn't see a way to run the integration tests with the main build 
>>>>> without a separate command.  Unless I can run everything in one 
>>>>> command, I'm pretty sure laziness will take over and I won't run 
>>>>> the integration tests.
>>>>>>
>>>>>>> I've made a
>>>>>>> possibly annoying change so that the default build includes IT.
 If
>>>>>>> you don't want them run
>>>>>>>
>>>>>>> mvn clean install -P no-it
>>>>>>>
>>>>>>> If this is too annoying we could reverse the profiles and have
the
>>>>>>> default leave out the it as before and a with-it profile that
>>>>>>> includes them.
>>>>>>
>>>>>> Yeah. I think the default profile should not run the IT. IMHO, I

>>>>>> think
>>>>>> it should not even run the unit tests by default. Developers (should)
>>>>>> run unit & IT tests before committing their code. And we have
>>>>>> automation builds with all tests that run 4 times a day anyways.
So
>>>>>> the default profile can well do away with tests. But that may just
be
>>>>>> my opinion.
>>>>> After working through some hard problem and getting ready to commit 
>>>>> the last thing anyone wants to do is remember a second command or 
>>>>> even a command line option to run some tests: they are apt to type 
>>>>> the simplest command that will check the build.  I think that 
>>>>> command should run all the tests, including the integration tests.  
>>>>> Even though there's enough time to go eat dinner while they run, 
>>>>> our most comprehensive checks will be run.  If you get too bored 
>>>>> you can always stop the build after you think enough stuff has been 
>>>>> checked, and if you remember you can run the build with options to 
>>>>> turn off whatever tests you want to skip.
>>>>> ApacheDS had their integration tests run using an option and most 
>>>>> people did not run them due to the extra effort of trying to 
>>>>> remember to type more on the command line.  I think they've changed 
>>>>> so the integration tests are run by default.
>>>>> I'm happy to keep talking about this.... and I'll be happy enough 
>>>>> with a "with-it" option, but I think running everything by default 
>>>>> is the best strategy.
>>>>>>
>>>>>>>
>>>>>>> Comments?
>>>>>>>
>>>>>>> This might have bad effects on Prasad's automation but I'm not
sure
>>>>>>> how that is run.
>>>>>>
>>>>>> For now, the automation builds have been modified to use the no-it

>>>>>> profile.
>>>>>>
>>>>> thanks!
>>>>> david jencks
>>>>>>>
>>>>>>> thanks && Happy New Year!
>>>>>>> david jencks
>>>>>>
>>>>>> Happy New Year to ya'll !
>>>>>> Prasad
> 
> 

Mime
View raw message