hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Meil <doug.m...@explorysmedical.com>
Subject Re: Running UnitTests before submitting a patch
Date Tue, 20 Sep 2011 14:22:57 GMT

Hi there-

https://issues.apache.org/jira/browse/HBASE-4448

I attached a prototype of HBaseTestingUtilityFactory to this ticket.  I'd
like to get some feedback on the general idea (emphasis on "general idea")
and then if the consensus is that it makes sense I will continue and then
we can do a more formal review on the reviews thing.

Doug




On 9/19/11 11:55 PM, "Stack" <stack@duboce.net> wrote:

>On Mon, Sep 19, 2011 at 1:06 PM, Jesse Yates <jesse.k.yates@gmail.com>
>wrote
>>> ... will spent (at least on my laptop) about 10.7 seconds setting up
>>>the
>>> cluster, and 7.3 seconds tearing down.  Assuming that we aren't
>>>running in
>>> a separate JVM each test invocation, sharing the same instance of
>>> HBaseTestingUtility would be an enormous benefit.
>>>
>
>Seems like it would be worth investigating makeing this run
>
>
>>> 1)  Re-use JVMs between test runs (if we are already doing this, then
>>>no
>>> action)
>>>
>>
>> This is already built into maven when we run tests normally. This leads
>>to
>> some interesting situations for messing around with class loading/global
>> variables; if you don't reset the environment, it can cause other
>> (essentially non-broken) tests to fail.  Unless we are running things in
>> forked mode .
>>
>> Forked mode could also cut down on the straight unit test time, but in
>>my
>> experience this has lead to difficulty in figuring out what crashed. For
>> some reason maven doesn't realize which tests failed, just that they
>>did, in
>> its final output (though it still stdouts "FAILURE").
>>
>
>Yeah, we run forked mode.
>
>          <configuration>
>            
><forkedProcessTimeoutInSeconds>900</forkedProcessTimeoutInSeconds>
>            <argLine>-enableassertions -Xmx1400m</argLine>
>            <redirectTestOutputToFile>true</redirectTestOutputToFile>
>          </configuration>
>
>I think the thinking was fork it because so much is going on and yeah,
>for sure there'll be cleanup afterward if all goes awry.
>
>Lohit suggested running tests in parallel.  I tried it but things
>didn't seem to run faster in my playing around.  See HBASE-4207.
>
>
>>> 2)  Re-use the same HBaseTestingUtility instance for all possible
>>>tests.
>>>
>>> I think #2 isn't "that bad" if we had a factory method to get/return
>>> HBaseTestingUtility instance and internally this factory could do a
>>> ref-count  and auto-detect of non-activity, then it could shutdown the
>>> cluster behind the scenes.
>>>
>>>
>> +1 on this idea.
>> We should also look into adding more/less region servers whenever people
>> request more. We could just leave it up for the entire span of the
>> integration tests and then bring it down afterwards (to make it easier
>>on
>> ourselves.
>>
>
>We do some of this now.   TestFromClientSide and TestAdmin are an
>aggregation of a bunch of tests that used to each spin up own
>clusters.
>
>We should do more of this.  I think what happens is that a bunch of
>our tests do one test only and to run the test, they spin up and down
>a cluster.  Could put a few together -- those that make sense.
>
>St.Ack


Mime
View raw message