hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-555) Only one Worker in HRS; on startup, if assigned tens of regions, havoc of reassignments because open processing is done in series
Date Mon, 31 Mar 2008 23:40:28 GMT

    [ https://issues.apache.org/jira/browse/HBASE-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583932#action_12583932
] 

stack commented on HBASE-555:
-----------------------------

Couple of ideas:

Create a worker thread for every message.  That'd be a worker per region to open.  If hundreds,
not so smart as all would be contending.  So perhaps an upper bound on threads created.  But
then we'd just have same issue again where we'd have queued opens that were not being serviced?

So maybe single Worker ain't so bad.  Issue then is making it so we report the master that
regions are being worked on.  Could take out an iterator and queue a PROCESSING message per
queued region.

Trying to come up w/ minimal patch for 0.1.  We can fix better in 0.2.

> Only one Worker in HRS; on startup, if assigned tens of regions, havoc of reassignments
because open processing is done in series
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-555
>                 URL: https://issues.apache.org/jira/browse/HBASE-555
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.16.0, 0.2.0, 0.1.0
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>
> On the Lars clusters, he's up into the thousands of regions.  Starting this cluster,
there is a load of churn in the master log as we assign regions, they report their opening
and then after the hbase.hbasemaster.maxregionopen of one minute elapses, we assign the region
elsewhere.
> Problem seems to be the fact that we only run a single Worker thread in our regionserver;
means that region opens are processed in series.
> For example, the below shows when a master assigned a region and then the regionserver
side log when it got around to opening it:
> {code}
> 2008-03-29 04:48:51,638 INFO org.apache.hadoop.hbase.HMaster: assigning region pdc-docs,US20060158177_20060720,1205765009844
to server 192.168.105.19:60020
> ..
> 2008-03-29 04:50:58,124 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN :
pdc-docs,US20060158177_20060720,1205765009844
> {code}
> There is > 2 minutes between the two loggings (I checked clocks on this cluster and
they are synced).
> Looking in the regionserver log, its just filled with logging on the opening of regions.
 The region opens are running pretty fast at about a second each but there are hundreds of
regions to open in this case so its easy to go over our default of 60 seconds.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message