hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chunhui shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5970) Improve the AssignmentManager#updateTimer and speed up handling opened event
Date Sat, 19 May 2012 11:38:08 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279526#comment-13279526
] 

chunhui shen commented on HBASE-5970:
-------------------------------------

bq.Why is doing this work in the background faster? Is it just that it being inline, it takes
a noticeable amount of time?

AssignmentManager#updateTimers would check each region in RIT, and there are lots of regions
in the RIT when startup, so it will took much time(about 30ms if 100k regions in RIT). However,
in the current logic, we will do the AssignmentManager#updateTimers for each opened event,
causing the whole process of handling all the opened events tooks much much time.

If we do the updateTimers in the background, we needn't wait it when handling opened event.
Also, we wouldn't updateTimers for the same Regionserver in a short time.(In fact, when cluster
startup , lots of opened events from the same regionserver at the same time, we needn't do
this work many times at the moment)
                
> Improve the AssignmentManager#updateTimer and speed up handling opened event
> ----------------------------------------------------------------------------
>
>                 Key: HBASE-5970
>                 URL: https://issues.apache.org/jira/browse/HBASE-5970
>             Project: HBase
>          Issue Type: Improvement
>          Components: master
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>         Attachments: 5970v3.patch, HBASE-5970.patch, HBASE-5970v2.patch, HBASE-5970v3.patch
>
>
> We found handing opened event very slow in the environment with lots of regions.
> The problem is the slow AssignmentManager#updateTimer.
> We do the test for bulk assigning 10w (i.e. 100k) regions, the whole process of bulk
assigning took 1 hours.
> 2012-05-06 20:31:49,201 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning
100000 region(s) round-robin across 5 server(s)
> 2012-05-06 21:26:32,103 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning
done
> I think we could do the improvement for the AssignmentManager#updateTimer: Make a thread
do this work.
> After the improvement, it took only 4.5mins
> 2012-05-07 11:03:36,581 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning
100000 region(s) across 5 server(s), retainAssignment=true 
> 2012-05-07 11:07:57,073 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning
done 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message