hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesse Yates <jesse.k.ya...@gmail.com>
Subject Re: scoping integration tests
Date Wed, 30 Nov 2011 06:28:44 GMT
+1 Overall idea.

I don't know if we should consider the integration tests as the primary
thing that bigtop runs. Since they are deploying to a small cluster
(right?), the tests that FB is supposed to be open sourcing soon should
probably be what they run (and lets call those acceptance tests).

While integration tests could be run by bigtop, they should definitely be
part of our testing of patches as well - no patches should go into trunk
that don't pass both the unit tests and integration tests. Then we push in
bigger patches or cut releases we make sure that the acceptance tests pass.

At least, that's what I was thinking about how testing works.

-Jesse

On Tue, Nov 29, 2011 at 10:08 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> From Stack:
> -------
> I think we can have it so everyone gets what they need;
> small/medium/large are done by surefire and then we have a new
> intergration test green field that is run by failsafe that can help us
> get tests for bigtop et al. to run.
>
> ------
>
> I think the above proposal makes sense.
>
> Please share your comments.
>
>
> On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> The following discussion is closely related to HBASE-4712.
>>
>> We should reach general consensus so that the execution of future test
>> strategy is smooth.
>>
>> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <jesse.k.yates@gmail.com>wrote:
>>
>>> I was considering 'integration tests' as a separate concern from the
>>> large/medium/small _unit_ tests.
>>>
>>> That is, in fact, why the failsafe plugin was added (and is designed
>>> for).
>>>
>>> Currently, we have a lot of tests that fall in the realm of integration
>>> tests (testing integration between various pieces, rather than single
>>> functionality alone). The most obvious indicator that something is an
>>> integration test is when it is using the MiniHBaseCluster since that is
>>> really testing the integration between the new feature and the rest of the
>>> system.
>>>
>>> The surefire plugin should really be testing unit tests, and then those
>>> can be classified as small/medium/large. However, the large would be
>>> _extremely_ rare cases.
>>>
>>> If the community is completely moving to just using the annotations for
>>> denoting size of tests, rather than using the integrationTest/unit test
>>> stuff, thats fine. However, I still think that if we do that, we should
>>> have a separate designation for @IntegrationTest (but that is really
>>> already covered in the fact that they would be named differently and
>>> therefore run by the different plugin). The reason we need to split them
>>> out is to have that separate tier of testing - once they pass unit level,
>>> they can be tested at the integration level and then on the acceptance test
>>> level (though we don't have any of those yet).
>>>
>>> This probably means more reclassification of tests.
>>>
>>> Making this explicit is important so people know at what point in the
>>> process their patch breaks (as well as staging the testing being good
>>> practice in general). Therefore, the small/medium/large classifications are
>>> not complete. Further, we should should still be able to use the general
>>> maven testing (mvn test, mvn verify) to run all the tests as expected,
>>> rather than having to know which things extra commands to run. To that end,
>>> we need to integrate HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>ASAP
so its in the book and people know how to work with the current
>>> testing layout.
>>>
>>> I've been a bit behind on the testing discussions on dev@, so didn't
>>> get to put these thoughts in, but I think they can still be added on. This
>>> was also part of what Doug, Stack and I were kind of envisioning in the
>>> original email to the list.
>>>
>>> Thanks.
>>>
>>> -Jesse
>>>
>>>
>>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nkeywal@gmail.com> wrote:
>>>
>>>> Today's tests are:
>>>>  - 416 small tests, executed in ~3 minutes
>>>>  - 489 medium tests, executed in ~35 minutes (without parallelization)
>>>>  - 280 large tests, executed in ~90 minutes (without parallelization)
>>>>
>>>> So may be you could put this new test as large? Will you want to run
>>>> this test for all central builds? Or do you see it as a specific tests that
>>>> will be executed only sometimes?
>>>>
>>>>
>>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nkeywal@gmail.com> wrote:
>>>>
>>>>> So I guess it's likely to be filtered as it's "not large".
>>>>>
>>>>>
>>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>>>
>>>>>> N:
>>>>>> Currently IntegrationTestConstraint doesn't have @Category annotation.
>>>>>>
>>>>>> FYI
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nkeywal@gmail.com>
wrote:
>>>>>>
>>>>>>> Yes, I put the large test as integration tests (may be 3 weeks
>>>>>>> ago?). So if you mix naming pattern with categories, you can
end up with no
>>>>>>> tests to do.
>>>>>>>
>>>>>>> It seems logical to me to put large tests in the failsafe module.
If
>>>>>>> you want to use the failsafe tests for something else, I can
remove then.
>>>>>>>
>>>>>>> see the doc in HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>(last
comment) and the patch
>>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847>
on
>>>>>>> how it designed to work today.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> N.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yuzhihong@gmail.com>
wrote:
>>>>>>>
>>>>>>>> Copying N.
>>>>>>>>
>>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
>>>>>>>> jesse.k.yates@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hmmm, that's odd. It was working last week, maybe it
was the
>>>>>>>>> recent changes N put in.
>>>>>>>>>
>>>>>>>>> I'll look into it.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> - Jesse Yates
>>>>>>>>>
>>>>>>>>> Sent from my iPhone.
>>>>>>>>>
>>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yuzhihong@gmail.com>
wrote:
>>>>>>>>>
>>>>>>>>> > Hi,
>>>>>>>>> > Please categorize new tests for 4605.
>>>>>>>>> >
>>>>>>>>> > When I issued this command:
>>>>>>>>> > mvn failsafe:integration-test
>>>>>>>>> >
>>>>>>>>> > I got:
>>>>>>>>> >
>>>>>>>>> > Results :
>>>>>>>>> >
>>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>>>>>>>>> >
>>>>>>>>> > Do you have some idea ?
>>>>>>>>> >
>>>>>>>>> > Thanks
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> -------------------
>>> Jesse Yates
>>> 240-888-2200
>>> @jesse_yates
>>>
>>>
>>
>


-- 
-------------------
Jesse Yates
240-888-2200
@jesse_yates

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message