hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2409) Daughters deployed to parents regionserver node, always
Date Mon, 05 Apr 2010 22:32:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853571#action_12853571

Jonathan Gray commented on HBASE-2409:

This kind of happens.

It's always been the case that daughters of a split region get assigned back to the original
regionserver.  In small clusters without lots of existing regions, this can cause the load
balancer to kick in and close the just opened daughters.

The story is kind of worse than that on a small cluster without lots of existing regions.

If you are writing fast enough, it's quite easy to get regions that are 4X or more the split
size.  This is because of stuff we're starting to deal with in other issues (HBASE-2375, double
flush optimization, etc).  By the time compaction finishes on a loaded cluster, you've managed
to snapshot/flush a bunch of additional StoreFiles for that same region.  So compaction finishes,
decides to split, we close the region, flush the memstore, split, regions get assigned back,
we open them, we compact them (we always compact opened regions when they contain references
because we do not support splitting splits).  When the daughters get compacted, we decide
to split again.  We close them, flush them, and again they get reassigned back.

We had cases on 5 node clusters where you would see one server hosting root, one server hosting
meta, two emtpy servers, and one server with 8 regions.

If you keep the regions in this loop where they decide to split post initial compaction, that
regionserver continues to get swamped.

Anyways, we clearly need to have one of the daughters go to a different rs :)  Would also
be good if it was not done in a fixed way that could cause the second daughter to always go
to the same (different) server.

> Daughters deployed to parents regionserver node, always
> -------------------------------------------------------
>                 Key: HBASE-2409
>                 URL: https://issues.apache.org/jira/browse/HBASE-2409
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.4, 0.21.0
> Ryan apparently said this 6 months ago, that on split, the regions are assigned back
to the parent hosting regionserver almost always.  Then, to keep up some kinda balance, the
load balancer kicks and closes the just opened daughters -- which have just been opened a
moment ago -- to deploy them elsewhere in the name of keeping good balance.
> This issue is to confirm the above behavior indeed happens and then to take action to
make it so at least one of the daughters is held up so it doesn't go back to the current heartbeating
host, the parent hosting server.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message