hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-555) Only one Worker in HRS; on startup, if assigned tens of regions, havoc of reassignments because open processing is done in series
Date Tue, 01 Apr 2008 19:48:24 GMT

    [ https://issues.apache.org/jira/browse/HBASE-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584274#action_12584274

stack commented on HBASE-555:

When .19 server shows up, gets 100+ regions.  This patch should include upper bound on how
many regions we assign at a time.

Looking at the Lars cluster, this patch seems to be doing what its supposed to.... Its ugly
in that currently there is a log for every MSG_REPORT_PROCESS_OPEN message -- one per region
every time it reports in -- but its just during startup (Previous our startup logs were clogged
with reassigning regions already assigned).

Here is illustration that patch is basically working...  After assignment, 7 seconds after
last assigned region message is logged we see this:

2008-04-01 00:07:05,460 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_PROCESS_OPEN
: pdc-docs,US20070223009_20070927,1205860531876 from

A few regions open, then we get this:

2008-04-01 00:07:11,528 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_PROCESS_OPEN
: pdc-docs,US4881767_19891121,1205704528908 from

... about 6 seconds after one from previous batch... then...

2008-04-01 00:07:14,534 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_PROCESS_OPEN
: pdc-docs,US5399923_19950321,1205793104251 from


2008-04-01 00:07:23,614 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_PROCESS_OPEN
: pdc-docs,EP04011653NWA1,1205771873299 from


> Only one Worker in HRS; on startup, if assigned tens of regions, havoc of reassignments
because open processing is done in series
> ---------------------------------------------------------------------------------------------------------------------------------
>                 Key: HBASE-555
>                 URL: https://issues.apache.org/jira/browse/HBASE-555
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.16.0, 0.2.0, 0.1.0
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>         Attachments: 555-0.1.patch
> On the Lars clusters, he's up into the thousands of regions.  Starting this cluster,
there is a load of churn in the master log as we assign regions, they report their opening
and then after the hbase.hbasemaster.maxregionopen of one minute elapses, we assign the region
> Problem seems to be the fact that we only run a single Worker thread in our regionserver;
means that region opens are processed in series.
> For example, the below shows when a master assigned a region and then the regionserver
side log when it got around to opening it:
> {code}
> 2008-03-29 04:48:51,638 INFO org.apache.hadoop.hbase.HMaster: assigning region pdc-docs,US20060158177_20060720,1205765009844
to server
> ..
> 2008-03-29 04:50:58,124 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN :
> {code}
> There is > 2 minutes between the two loggings (I checked clocks on this cluster and
they are synced).
> Looking in the regionserver log, its just filled with logging on the opening of regions.
 The region opens are running pretty fast at about a second each but there are hundreds of
regions to open in this case so its easy to go over our default of 60 seconds.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message