hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15207) Stuck balancer
Date Wed, 03 Feb 2016 07:09:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129933#comment-15129933
] 

stack commented on HBASE-15207:
-------------------------------

Happened again in new loading. Its hard to make sense of because the logging overwhelms. Let
me address that in a patch first. Can then look at hang. In this current case we filled 10
log files of 256MB each... but it looks like we 'recovered'. LB reports:

2016-02-02 17:15:05,123 DEBUG [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
balancer.StochasticLoadBalancer: Finished computing new load balance plan.  Computation took
3761ms to try 217600 different iterations.  Found a solution that moves 31 regions; Going
from a computed cost of 341.5035724035313 to a new cost of 47.8518130393211

And going back over logs, yeah, I get sessions of log spewing.... maybe the load balancer
is ok... its just this crazy logging phenomenon




> Stuck balancer
> --------------
>
>                 Key: HBASE-15207
>                 URL: https://issues.apache.org/jira/browse/HBASE-15207
>             Project: HBase
>          Issue Type: Bug
>          Components: Balancer
>    Affects Versions: 1.2.0
>            Reporter: stack
>
> Balancer seems to have gotten stuck in 1.2.0RC1 soon after Master joins running cluster
(previous Master had been killed by chaos monkey). Investigate. At least fix the crazy logging
which made me notice the stuck balancer.
> Last night my logs filled with this (10x256MB log files):
> ....
> 2016-02-01 11:25:26,958 DEBUG [B.defaultRpcServer.handler=9,queue=0,port=16000] balancer.BaseLoadBalancer:
Lowest locality region server with non zero regions is ve0542.halxg.cloudera.com with locality
0.0
> 2016-02-01 11:25:26,958 DEBUG [B.defaultRpcServer.handler=9,queue=0,port=16000] balancer.BaseLoadBalancer:
 Lowest locality region index is 0 and its region server contains 1 regions
> ...
> Added by this:
> commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
> Author: tedyu <yuzhihong@gmail.com>
> Date:   Thu Aug 13 09:11:59 2015 -0700
>     HBASE-13376 Improvements to Stochastic load balancer (Vandana Ayyalasomayajula)
> Looks like balancer got stuck. Logging at ten lines a millisecond.
> Here is lead up. Nothing in particular jumps out. Rerun doesn't show this.
> {code}
> 2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions.
> 2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions.
> 2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions.
> 2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Lowest locality region server with non zero regions is ve0540.halxg.cloudera.com
with locality 0.0
> 2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer:  Lowest locality region index is 0 and its region server contains
1 regions
> 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions.
> 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions.
> 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions.
> 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Lowest locality region server with non zero regions is ve0540.halxg.cloudera.com
with locality 0.0
> 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer:  Lowest locality region index is 0 and its region server contains
1 regions
> 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions.
> 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions.
> 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions.
> ....
> {code}
> Nothing else is happening on this master
> Happens just after a Master joins cluster after being killed by a monkey.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message