hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chunhui shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6134) Speed up distribued split log
Date Wed, 30 May 2012 08:23:23 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285496#comment-13285496
] 

chunhui shen commented on HBASE-6134:
-------------------------------------

@ted

In my opition, we could do the improvement for split-worker.


In the patch , I change the split logic which executed by the split-worker same as local-master-split.

It means, there are multi threads to creater writer and append hlog entry.

Also, we found closing writer is quite slow, so make the closeStreams() executed by multi
threads.

With the patch, we test the time of distribued split log again, and reduce the time from 165s+
to 38s+
(34hlog files 4 regionervers)
                
> Speed up distribued split log
> -----------------------------
>
>                 Key: HBASE-6134
>                 URL: https://issues.apache.org/jira/browse/HBASE-6134
>             Project: HBase
>          Issue Type: Improvement
>          Components: wal
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>             Fix For: 0.96.0
>
>         Attachments: HBASE-6134.patch
>
>
> First,we do the test between local-master-split and distributed split log
> Environment:34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting
work)
> local-master-split:60s+
> distributed-split-log:165s+
> In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers
for 34 hlog files (regionserver may be in high load)
> We found split-worker split one log file took about 20s.
> I think we should do the improvement for this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message