hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From N Keywal <nkey...@gmail.com>
Subject Re: FW: Speeding up tests
Date Mon, 17 Oct 2011 12:30:58 GMT

I will be working for a month on the subject, on behalf of StumbleUpon /
Stack. The goal is to reduced the build time for developer to a minimum, and
at least half of the time needed now (i.e: from two hours -> 1 hour).

I created a JIRA to ease the follow up: HBASE-4602. I will put all the
future sub-JIRA in this one. I already put the existing ones as "related

As a start, I extracted the time taken on the apache server today, plus some
hints on what the test is doing: the type of cluster used (dfs, zookeeper,
hbase, mapreduce), the logs, potential "Thread.sleep". I attach the
resulting excel sheet in HBASE-4602, you may want to have a look. BTW, The
second sheet contains the script I used for this.

Strategy will be mainly:
- Cutting down on the number of cluster spinups by coalescing related tests
rather than have each spin up its own cluster
- Make cluster start/stop faster
- Rewriting long-running tests so they do not need to be run on a cluster;
e.g. by instead mocking expected signals/messages
- Move long running tests out of the unit test suite to instead run as part
of the recently introduced 'integration test' step

Of course, there will be numerous small JIRAs to avoid any big bang effect.

Splitting the tests in unit tests vs. long tests seems quite promising when
looking to the excel sheet. Jesse, I understood that you're already working
on this? Will you do the split as well?

For myself, at the beginning, I will concentrate on cleaning up the tests
and improving the start time of the cluster, so you will see some JIRA on
this. Then I will look at  the "long tests" that we would really like to
keep as "unit test".



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message