hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek Singh Chouhan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17791) Locality should not be affected for non-faulty region servers at startup
Date Fri, 31 Mar 2017 00:14:42 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950094#comment-15950094
] 

Abhishek Singh Chouhan commented on HBASE-17791:
------------------------------------------------

In HBASE-15251 failover is set to false even when we get an non empty list of deadServers
{code}
if (failover) {
      // This may not be a failover actually, especially if meta is on this master.
      if (LOG.isDebugEnabled()) {
        LOG.debug("Found dead servers out on cluster " + serverManager.getDeadServers());
      }
      // Check if there are any regions on these servers
      failover = false;
....
{code}

Now this is set to true only when one of the many checks like region except meta in transition
etc. are true which look like valid scenarios , otherwise it stays false and we go ahead with
assigning regions as per meta. Won't this already solve the problem ?



> Locality should not be affected for non-faulty region servers at startup
> ------------------------------------------------------------------------
>
>                 Key: HBASE-17791
>                 URL: https://issues.apache.org/jira/browse/HBASE-17791
>             Project: HBase
>          Issue Type: Improvement
>          Components: Balancer, Region Assignment
>    Affects Versions: 1.3.0, 1.0.3, 1.2.4, 1.1.8
>            Reporter: Lars George
>            Priority: Blocker
>
> We seem to have an issue with store file locality as soon as a single server is missing
or faulty upon restart. HBASE-15251 is addressing a subset of the problem, ensuring that some
remaining files or an empty server do not trigger a full recovery, but is missing further,
similar cases. Especially, the case where a server fails to report in before the active master
is finished waiting for them to do so, or where servers have been decomissioned (but not removed
from the {{regionservers}} file), and finally the true case of a dead node.
> In case a single node is faulty, the user regions are _not_ assigned as saved in the
{{hbase:meta}} table, but completely randomized in a round-robin fashion. An additional factor
is that in this case the regions are _not_ assigned to the best matching node (the one with
a copy of the data locally), but to any node, leaving the locality in shambles.
> What is also bad, if the {{hbase.hstore.min.locality.to.skip.major.compact}} property
is left at the default {{0.0f}}, then an older region that had no writes since the last major
compaction happened is just skipped (as expected, usually) and locality stays bad as-is. All
reads for those aged-out regions will be network reads. But in any event, having to run a
major compaction after a restart is not good anyways.
> The issue is the code in {{AssignmentManager.processDeadServersAndRegionsInTransition()}},
which is handed a list of dead servers. But it immediately sets the {{failover}} flag and
the code
> {code}
>     failoverCleanupDone();
>     if (!failover) {
>       // Fresh cluster startup.
>       LOG.info("Clean cluster startup. Don't reassign user regions");
>       assignAllUserRegions(allRegions);
>     } else {
>       LOG.info("Failover! Reassign user regions");
>     }
> {code}
> is not triggering the assignment of the regions to those servers that are still present
and have all their region data local. What should happen is that only the missing regions
are reassigned, just like in the case of a server failing while the cluster is running.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message