hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Marc Spaggiari (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10592) Refactor PerformanceEvaluation tool
Date Wed, 26 Feb 2014 21:34:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913540#comment-13913540
] 

Jean-Marc Spaggiari commented on HBASE-10592:
---------------------------------------------

Works well for me in 0.96
With your patch:
{code}
hbase@hbasetest1:~$  bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --presplit=6
--rows=2097152 randomWrite 1
2014-02-26 15:22:28,980 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated.
Instead, use io.native.lib.available
2014-02-26 15:22:29,799 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x52c03865
connecting to ZooKeeper ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:22:30,360 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5814caa8
connecting to ZooKeeper ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:22:30,391 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x5814caa8
connecting to ZooKeeper ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:22:32,137 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x5814caa8
connecting to ZooKeeper ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:22:32,190 INFO  [main] hbase.PerformanceEvaluation: Table created with 6 splits
2014-02-26 15:22:32,197 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x5814caa8
connecting to ZooKeeper ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:22:32,221 INFO  [main] hbase.PerformanceEvaluation: Start class org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest
at offset 0 for 2097152 rows
2014-02-26 15:22:32,298 INFO  [main] hbase.PerformanceEvaluation: Timed test starting in thread
main
2014-02-26 15:22:52,933 INFO  [main] hbase.PerformanceEvaluation: 0/209715/2097152
2014-02-26 15:23:06,678 INFO  [main] hbase.PerformanceEvaluation: 0/419430/2097152
2014-02-26 15:23:22,174 INFO  [main] hbase.PerformanceEvaluation: 0/629145/2097152
2014-02-26 15:23:39,249 INFO  [main] hbase.PerformanceEvaluation: 0/838860/2097152
2014-02-26 15:23:54,392 INFO  [main] hbase.PerformanceEvaluation: 0/1048575/2097152
2014-02-26 15:24:07,971 INFO  [main] hbase.PerformanceEvaluation: 0/1258290/2097152
2014-02-26 15:24:26,551 INFO  [main] hbase.PerformanceEvaluation: 0/1468005/2097152
2014-02-26 15:24:38,495 INFO  [main] hbase.PerformanceEvaluation: 0/1677720/2097152
2014-02-26 15:24:51,138 INFO  [main] hbase.PerformanceEvaluation: 0/1887435/2097152
2014-02-26 15:25:09,124 INFO  [main] hbase.PerformanceEvaluation: 0/2097150/2097152
2014-02-26 15:25:09,360 INFO  [main] hbase.PerformanceEvaluation: Finished class org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest
in 157058ms at offset 0 for 2097152 rows (13,17 MB/s)
2014-02-26 15:25:09,360 INFO  [main] client.HConnectionManager$HConnectionImplementation:
Closing master protocol: MasterService
2014-02-26 15:25:09,360 INFO  [main] client.HConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x14426a37e621805
{code}

Without the patch
{code}
hbase@hbasetest1:~$ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --presplit=6 --rows=2097152
randomWrite 1
2014-02-26 15:29:57,987 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated.
Instead, use io.native.lib.available
2014-02-26 15:29:58,663 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
2014-02-26 15:29:58,832 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x545aae15
connecting to ZooKeeper ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:29:59,400 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4f7540bd
connecting to ZooKeeper ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:29:59,436 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x4f7540bd
connecting to ZooKeeper ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:30:00,665 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x4f7540bd
connecting to ZooKeeper ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:30:00,706 INFO  [main] hbase.PerformanceEvaluation: Table created with 6 splits
2014-02-26 15:30:00,719 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x4f7540bd
connecting to ZooKeeper ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:30:00,765 INFO  [main] hbase.PerformanceEvaluation: Start class org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest
at offset 0 for 2097152 rows
2014-02-26 15:30:00,827 INFO  [main] hbase.PerformanceEvaluation: Timed test starting in thread
main
2014-02-26 15:30:20,306 INFO  [main] hbase.PerformanceEvaluation: 0/209715/2097152
2014-02-26 15:30:34,670 INFO  [main] hbase.PerformanceEvaluation: 0/419430/2097152
2014-02-26 15:30:48,351 INFO  [main] hbase.PerformanceEvaluation: 0/629145/2097152
2014-02-26 15:31:04,139 INFO  [main] hbase.PerformanceEvaluation: 0/838860/2097152
2014-02-26 15:31:19,112 INFO  [main] hbase.PerformanceEvaluation: 0/1048575/2097152
2014-02-26 15:31:31,711 INFO  [main] hbase.PerformanceEvaluation: 0/1258290/2097152
2014-02-26 15:31:50,578 INFO  [main] hbase.PerformanceEvaluation: 0/1468005/2097152
2014-02-26 15:32:03,861 INFO  [main] hbase.PerformanceEvaluation: 0/1677720/2097152
2014-02-26 15:32:16,284 INFO  [main] hbase.PerformanceEvaluation: 0/1887435/2097152
2014-02-26 15:32:32,913 INFO  [main] hbase.PerformanceEvaluation: 0/2097150/2097152
2014-02-26 15:32:33,245 INFO  [main] hbase.PerformanceEvaluation: Finished class org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest
in 152415ms at offset 0 for 2097152 rows (13,57 MB/s)
2014-02-26 15:32:33,245 INFO  [main] client.HConnectionManager$HConnectionImplementation:
Closing master protocol: MasterService
2014-02-26 15:32:33,246 INFO  [main] client.HConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x14426a37e621818
{code}

Similar output, similar result. Looked at the code too.
+1

> Refactor PerformanceEvaluation tool
> -----------------------------------
>
>                 Key: HBASE-10592
>                 URL: https://issues.apache.org/jira/browse/HBASE-10592
>             Project: HBase
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.96.2, 0.98.1, 0.99.0
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>            Priority: Minor
>             Fix For: 0.99.0
>
>         Attachments: HBASE-10592.00-0.96.patch, HBASE-10592.00-0.98.patch, HBASE-10592.00.patch,
HBASE-10592.01-0.96.patch, HBASE-10592.01-0.98.patch, HBASE-10592.01.patch
>
>
> PerfEval is kind of a mess. It's painful to add new features because the test options
are itemized and passed as parameters to internal methods. Serialization is hand-rolled and
tedious. Ensuring support for mapreduce mode is a chore because of it.
> This patch refactors the tool. Options are now passed around to methods and such as a
POJO instead of one-by-one. Get rid of accessors that don't help anyone. On the mapreduce
side, serialization is now handled using json (jackson is a dependency anyway) instead of
the hand-rolled regex we used before. Also do away with custom InputSplit and FileFormat,
instead using Text and NLineInputFormat. On the local mode side, combine 1 client and N clients
into the same implementation. That implementation now uses an ExecutorService, so we can later
decouple number of client workers from number of client tasks. Finally, drop a bunch of confusing
local state, instead use the new TestOptions POJO as a parameter to static methods.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message