hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4908) HBase cluster test tool (port from 0.89-fb)
Date Wed, 07 Dec 2011 02:33:42 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164079#comment-13164079
] 

Phabricator commented on HBASE-4908:
------------------------------------

mbautin has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from
0.89-fb)".

  Thanks for reviews, Nicolas and Stack! See responses below. A new version of the code will
follow. Still have to re-run the unit tests (I've been having some trouble with those recently
-- http://pastebin.com/1G1ZcPeV) and sanity-check the command-line load tester. On a side
note, here are some stats from my recent load test run on a 5-node cluster:

  11/12/06 18:30:25 INFO util.MultiThreadedAction: [W:21] Keys=17180542, cols=819.4m, time=27:49:20
Overall: [keys/s= 171, latency=116 ms] Current: [keys/s=219, latency=90 ms], insertedUpTo=17180495,
insertedQSize=26
  11/12/06 18:30:25 INFO util.MultiThreadedAction: [R:10] Keys=250690067, cols=11.8g, time=27:49:20
Overall: [keys/s= 2502, latency=3 ms] Current: [keys/s=261, latency=38 ms], verified=250690067

  (The number of writer's threads is reported as 21 because there is an "inserted keys tracker"
thread that keeps track of the most recent contiguous key written by all writers.)

INLINE COMMENTS
  src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:129 Changed this to
80. There is no standard way to get terminal width in Java. http://stackoverflow.com/questions/1286461/can-i-find-the-console-width-with-java

  src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:40 This is the command-line part
of LoadTest, so it should not run as part of the test suite. The same multithreaded writer/verifier
code is reused in a couple of "large" unit tests, namely TestMiniClusterLoad{Parallel,Sequential}.

  I added a line about the differences between this load tester and PerformanceEvaluation.

  Renamed this to LoadTestTool.
  src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:318 We first wait for all writers
to finish and then wait for all readers to finish, so we should exit when all of those threads
stop.
  src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java:37 Yes, this can be converted
to an IntegrationTest. I think making this a unit test was proposed in the past but the controversy
was that it spawns a bunch of child processes and extra care should be taken to shut down
all of them. We can leave this as a command-line tool for now and convert to a unit test when
it is more stable.
  src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1658-1680 Removed these methods and
reused the existing RegionSetting.HexStringSplit class.
  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java:28 I moved this to from the test
jar to the main jar because of the waitForBaseZNode method that I added to ZKUtil. We need
this kind of watcher for quick ZK checks when we don't want to use a callback. Passing a null
instead of a watcher does not work.
  src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java:176 Not sure what
changes you are talking about. I made a few conf options that I needed to use constants in
HConstants.
  src/main/java/org/apache/hadoop/hbase/util/Keying.java:41-43 Removed this.
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1820-1821 Done. Thanks!
  src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:39 Renamed this to AbstractHBaseTool
and moved to the main jar, since this is general-purpose command line processing functionality.
This also avoids the confusion with the IntegrationTest notation (thanks for the link!)  I
think we will be creating a lot of integration tests very soon.

REVISION DETAIL
  https://reviews.facebook.net/D549

                
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
>                 Key: HBASE-4908
>                 URL: https://issues.apache.org/jira/browse/HBASE-4908
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>         Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch,
D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator
and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has
some features that I have not tried yet (some kind of a kill test, and some way to run HBase
as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line
tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node
dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test
-read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but
I am submitting the patch for early feedback. I will probably try out its other functionality
and comment on how it works.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message