hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kannan Muthukkaruppan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed-split-log
Date Thu, 31 May 2012 15:04:23 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286638#comment-13286638
] 

Kannan Muthukkaruppan commented on HBASE-6134:
----------------------------------------------

The SequenceFile writer does its own buffering. And so, my guess is that, at close time, the
flushing of these buffers for every region's recovered.edits (SequenceFile) file is what is
causing the close to be slow because it is serialized on these flushes and then DN reporting
back to NN that it has received the blocks. Parallelizing the close in threads makes sense.

Good catch Chunhui! May I request you to update the description for the JIRA with a summary
of what the fix does, so it is useful for future readers of this JIRA.
                
> Improvement for split-worker to speed up distributed-split-log
> --------------------------------------------------------------
>
>                 Key: HBASE-6134
>                 URL: https://issues.apache.org/jira/browse/HBASE-6134
>             Project: HBase
>          Issue Type: Improvement
>          Components: wal
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>            Priority: Critical
>             Fix For: 0.96.0
>
>         Attachments: HBASE-6134.patch, HBASE-6134v2.patch, HBASE-6134v3.patch
>
>
> First,we do the test between local-master-split and distributed split log
> Environment:34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting
work)
> local-master-split:60s+
> distributed-split-log:165s+
> In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers
for 34 hlog files (regionserver may be in high load)
> We found split-worker split one log file took about 20s.
> I think we should do the improvement for this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message