hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From N Keywal <nkey...@gmail.com>
Subject Re: FW: Speeding up tests
Date Mon, 17 Oct 2011 20:20:20 GMT
Hi Jesse,

Yes, sounds good! Let me rephrase to be sure I understood you point: you're
saying that sometimes the test has a value only if there are some real
components behind, and mocking would make the test not very useful?

I agree with this, and I will first try to speed up the tests globally, then
attack the remaining issues, with mocking as a possible solution but not a
silver bullet to use without distinction.

I was also thinking about splitting the test 'as they are' between the long
running ones & and the others, without changing them. I put a possible list
in the column 'P' of the excel sheet in HR-4602. The long running ones list
is in mine mind something as: slow test AND (flaky test OR very specialized
test OR test interesting only with a third party). A test that fails often
(i.e. a good bug catcher) does not fall in this category. This allow to have
valuable tests with a set of test that every developer should run before
submitting a bug. What do you think?

May be it's possible with Jenkins to run the two sets in two JVM in // as
well (haven't check yet), so on a build server all the test could be run
efficiently as well.

This said, on the very short term, I am concentrating on making the
MiniCluster starts & stops faster... See HBase-4603 for example.

And I will add a link to 4438 before doing any split!

Cheers,



On Mon, Oct 17, 2011 at 7:40 PM, Jesse Yates <jesse.k.yates@gmail.com>wrote:

> Keywal,
>
> Thanks for helping out with this.
>
> Yeah, I've started working on breaking out some of the tests from unit to
> integration tests (see HBASE-4559). Basically, I'm just working from top to
> bottom on the source tree, trying to pull out integration tests and, when
> possible, replace some of the testing with unit tests backed by mocking.
>
> The unit test version wasn't really possible with 4559 as all the avro
> stuff
> would essentially making sure that it makes the one or two calls to a
> cluster with essentially no transformation. This kind of thing is not
> really
> worth unit testing as there is no internal behavior being tested, but would
> instead just mean mocking all the internals. A unit test would tell you:
> "yeah, it calls these methods in this order," but it is going to break as
> soon as any behavior changes in the class under test.
>
> The reason I mention the above is I would caution against writing a unit
> test that does all this internal mocking; it is a false comfort in that the
> test passes because you made it so, not really because the functionality is
> truly "correct", meaning it is actuall worse than not having the unit test
> and just relying on the integration test.
>
> That being said, I'm excited to have you help out with this effort. As a
> way
> to make sure we don't overlap work, just make sure you add a ticket for
> split a test/package AND link it to 4438 (the original umbrella ticket)
> _before_ spending time on doing the extraction.
>
> Sound good?
>
> -Jesse
>
> On Mon, Oct 17, 2011 at 5:30 AM, N Keywal <nkeywal@gmail.com> wrote:
>
> > Hello,
> >
> > I will be working for a month on the subject, on behalf of StumbleUpon /
> > Stack. The goal is to reduced the build time for developer to a minimum,
> > and
> > at least half of the time needed now (i.e: from two hours -> 1 hour).
> >
> > I created a JIRA to ease the follow up: HBASE-4602. I will put all the
> > future sub-JIRA in this one. I already put the existing ones as "related
> > link".
> >
> > As a start, I extracted the time taken on the apache server today, plus
> > some
> > hints on what the test is doing: the type of cluster used (dfs,
> zookeeper,
> > hbase, mapreduce), the logs, potential "Thread.sleep". I attach the
> > resulting excel sheet in HBASE-4602, you may want to have a look. BTW,
> The
> > second sheet contains the script I used for this.
> >
> > Strategy will be mainly:
> > - Cutting down on the number of cluster spinups by coalescing related
> tests
> > rather than have each spin up its own cluster
> > - Make cluster start/stop faster
> > - Rewriting long-running tests so they do not need to be run on a
> cluster;
> > e.g. by instead mocking expected signals/messages
> > - Move long running tests out of the unit test suite to instead run as
> part
> > of the recently introduced 'integration test' step
> >
> > Of course, there will be numerous small JIRAs to avoid any big bang
> effect.
> >
> > Splitting the tests in unit tests vs. long tests seems quite promising
> when
> > looking to the excel sheet. Jesse, I understood that you're already
> working
> > on this? Will you do the split as well?
> >
> > For myself, at the beginning, I will concentrate on cleaning up the tests
> > and improving the start time of the cluster, so you will see some JIRA on
> > this. Then I will look at  the "long tests" that we would really like to
> > keep as "unit test".
> >
> >
> > Regards,
> >
> > N.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message