hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-3871) Speedup LoadIncrementalHFiles by parallelizing HFile splitting
Date Tue, 10 May 2011 18:41:47 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031299#comment-13031299

Ted Yu commented on HBASE-3871:

To answer question number 2 above, allow me to reference what Adam said (the patch refers
to the one in HBASE-3721):

Ok, I did a number of runs of loading a single set of HFiles with and without the patch, and
it does seem the patch improves the load speed. I'll need to run more extensively to get accurate
numbers, but with the patch I'm seeing ranges from 3-7 minutes vs 5-11 without the patch.

> Speedup LoadIncrementalHFiles by parallelizing HFile splitting
> --------------------------------------------------------------
>                 Key: HBASE-3871
>                 URL: https://issues.apache.org/jira/browse/HBASE-3871
>             Project: HBase
>          Issue Type: Improvement
>          Components: mapreduce
>    Affects Versions: 0.90.2
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>         Attachments: 3871.patch
> From Adam w.r.t. HFile splitting:
> There's actually a good number of messages of that type (HFile no longer fits inside
a single region), unfortunately I didn't take a timestamp on just when I was running with
the patched jars vs the regular ones, however from the logs I can say that this is occurring
fairly regularly on this system.  The cluster I tested this on is our backup cluster, the
mapreduce jobs on our production cluster output HFiles which are copied to the backup and
then loaded into HBase on both.  Since the regions may be somewhat different on the backup
cluster I would expect it to have to split somewhat regularly.
> This JIRA complements HBASE-3721 by parallelizing HFile splitting which is done in the
main thread.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message