hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4916) LoadTest MR Job
Date Thu, 22 Dec 2011 23:35:31 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175152#comment-13175152
] 

Phabricator commented on HBASE-4916:
------------------------------------

jdcryans has commented on the revision "HBASE-4916 [jira] LoadTest MR Job".

  Another day of testing later, here's what I think of the current patch:

   - I like being able to drive a lot of load from just a few mappers, this is definitely
something YCSB and PE can't do.
   - Being able to mix things is fun too, with PE I'd have to start 2 tests. With YCSB it's
more versatile tho since you can tweak more than just random reads and writes.
   - Regarding the work loads, it's also much easier to do in YCSB as you don't have to write
code.

  My other comments are in the code below.

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadTest.java:197 HBC.create()  ??
  src/main/java/org/apache/hadoop/hbase/loadtest/Workload.java:100 I understand why the table
is recreated every time (you need to know the existing keys), but this is also a big flaw
for this load tester. If you have a respectable number of nodes, it's takes a lot of time
to generate enough data so that your block cache/OS cache cannot hold all the data. Being
able to restart using an existing table is a must have IMO.

  How are you guys using it?
  src/main/java/org/apache/hadoop/hbase/loadtest/Workload.java:136 I think this should be
configurable.
  src/main/java/org/apache/hadoop/hbase/loadtest/GetGenerator.java:97 This is one of the other
big problem with this load generator, the only pattern is random. YCSB let's you do that but
also offers zipf or latest distribution.

REVISION DETAIL
  https://reviews.facebook.net/D741

                
> LoadTest MR Job
> ---------------
>
>                 Key: HBASE-4916
>                 URL: https://issues.apache.org/jira/browse/HBASE-4916
>             Project: HBase
>          Issue Type: Sub-task
>          Components: client, regionserver
>            Reporter: Nicolas Spiegelberg
>            Assignee: Christopher Gist
>             Fix For: 0.94.0
>
>         Attachments: HBASE-4916.D741.1.patch
>
>
> Add a script to start a streaming map-reduce job where each map tasks runs an instance
of the load tester for a partition of the key-space. Ensure that the load tester takes a parameter
indicating the start key for write operations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message