hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4916) LoadTest MR Job
Date Thu, 22 Dec 2011 23:35:31 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175152#comment-13175152

Phabricator commented on HBASE-4916:

jdcryans has commented on the revision "HBASE-4916 [jira] LoadTest MR Job".

  Another day of testing later, here's what I think of the current patch:

   - I like being able to drive a lot of load from just a few mappers, this is definitely
something YCSB and PE can't do.
   - Being able to mix things is fun too, with PE I'd have to start 2 tests. With YCSB it's
more versatile tho since you can tweak more than just random reads and writes.
   - Regarding the work loads, it's also much easier to do in YCSB as you don't have to write

  My other comments are in the code below.

  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadTest.java:197 HBC.create()  ??
  src/main/java/org/apache/hadoop/hbase/loadtest/Workload.java:100 I understand why the table
is recreated every time (you need to know the existing keys), but this is also a big flaw
for this load tester. If you have a respectable number of nodes, it's takes a lot of time
to generate enough data so that your block cache/OS cache cannot hold all the data. Being
able to restart using an existing table is a must have IMO.

  How are you guys using it?
  src/main/java/org/apache/hadoop/hbase/loadtest/Workload.java:136 I think this should be
  src/main/java/org/apache/hadoop/hbase/loadtest/GetGenerator.java:97 This is one of the other
big problem with this load generator, the only pattern is random. YCSB let's you do that but
also offers zipf or latest distribution.


> LoadTest MR Job
> ---------------
>                 Key: HBASE-4916
>                 URL: https://issues.apache.org/jira/browse/HBASE-4916
>             Project: HBase
>          Issue Type: Sub-task
>          Components: client, regionserver
>            Reporter: Nicolas Spiegelberg
>            Assignee: Christopher Gist
>             Fix For: 0.94.0
>         Attachments: HBASE-4916.D741.1.patch
> Add a script to start a streaming map-reduce job where each map tasks runs an instance
of the load tester for a partition of the key-space. Ensure that the load tester takes a parameter
indicating the start key for write operations.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message