hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-3721) Speedup LoadIncrementalHFiles
Date Fri, 29 Apr 2011 20:51:03 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027178#comment-13027178
] 

jiraposter@reviews.apache.org commented on HBASE-3721:
------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/572/
-----------------------------------------------------------

(Updated 2011-04-29 20:48:41.082584)


Review request for hbase and Todd Lipcon.


Changes
-------

Simplified the changes for this JIRA according to Todd's review.
TestLoadIncrementalHFiles and TestHFileOutputFormat pass.


Summary
-------

I refactored LoadIncrementalHFiles so that tryLoad() queues work items in List<ServerCallable<Void>>.
doBulkLoad() periodically sends batch of ServerCallable's to HBase cluster.
I added the following method to HConnection/HConnectionManager:
    public <T> void getRegionServerWithRetries(ExecutorService pool,
        List<ServerCallable<T>> callables, Object[] results)
This method uses thread pool to send multiple ServerCallable's through getRegionServerWithRetries(ServerCallable<T>
callable).

I introduced two new config parameters: hbase.loadincremental.threads.max and hbase.loadincremental.batch.size
hbase.loadincremental.batch.size is for configuring the batch size above which HConnection.getRegionServerWithRetries()
would be called. In Adam's case, there're many small HFiles. LoadIncrementalHFiles shouldn't
wait until all HFiles have been scanned.
hbase.loadincremental.threads.max controls the maximum number of threads in thread pool.


This addresses bug HBASE-3721.
    https://issues.apache.org/jira/browse/HBASE-3721


Diffs (updated)
-----

  /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1097897 

Diff: https://reviews.apache.org/r/572/diff


Testing
-------

TestLoadIncrementalHFiles and TestHFileOutputFormat pass.


Thanks,

Ted



> Speedup LoadIncrementalHFiles
> -----------------------------
>
>                 Key: HBASE-3721
>                 URL: https://issues.apache.org/jira/browse/HBASE-3721
>             Project: HBase
>          Issue Type: Improvement
>          Components: util
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>         Attachments: 3721-v2.txt, 3721-v3.txt, 3721-v4.txt, 3721.txt
>
>
> From Adam Phelps:
> from the logs it looks like <1% of the hfiles we're loading have to be split.  Looking
at the code for LoadIncrementHFiles (hbase v0.90.1), I'm actually thinking our problem is
that this code loads the hfiles sequentially.  Our largest table has over 2500 regions and
the data being loaded is fairly well distributed across them, so there end up being around
2500 HFiles for each load period.  At 1-2 seconds per HFile that means the loading process
is very time consuming.
> Currently server.bulkLoadHFile() is a blocking call.
> We can utilize ExecutorService to achieve better parallelism on multi-core computer.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message