hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaolin Ha (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized
Date Thu, 08 Feb 2018 14:57:00 GMT

     [ https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Xiaolin Ha updated HBASE-19937:
-------------------------------
    Description: 
hbase:rsgroup table will be created by calling createRSGroupTable when master load system
coprocessors in 

 
{code:java}
844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
when ProcedureExecutor hasn't been initialized before createRSGroupTable, it will encounter
Exception as follows,

 
{noformat}
Exception in thread "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
java.lang.IllegalArgumentException
at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
while ProcedureExecutor  initialized by calling
{code:java}
848  startServiceThreads();{code}
And LoadBalancer is initialized by calling 
{code:java}
868  this.balancer.initialize();{code}
When LoadBalancer hasn't been initialized before createRSGroupTable, it will encounters
Exception as follows,

2018-02-02,16:12:45,688 ERROR org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG:
Uncaught runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure
table=hbase:rsgroup
 java.lang.NullPointerException
 at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
 at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
 at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
 at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
 at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
 at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
 at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)

 

As a result of CreateTableProcedure.rollbackState, it may then print logs warning TableExistsException
as follows,

2018-02-02,16:12:55,503 WARN org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
Failed to perform check
 java.io.IOException: Failed to create group table. org.apache.hadoop.hbase.TableExistsException:
hbase:rsgroup
 at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)

 

After some auto-retries, it loops running the thread RSGroupStartupWorker, will print logs
as follows, 

2018-02-02,16:23:17,626 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
 2018-02-02,16:23:17,730 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
 2018-02-02,16:23:17,834 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
 2018-02-02,16:23:17,937 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true

 

And using shells of rsgroup, it will tips that currently is in "offline mode".

 

The reason of this problem is that the order of createRSGroupTable and initializing of ProcedureExecutor
and LoadBalancer is out of control. If the former is excuted earlier, it will encounter
Exception mentioned before.

 

  was:
rsgroup table will be created by calling createRSGroupTable when master load system coprocessors
in 

 
{code:java}
844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
when ProcedureExecutor hasn't been initialized before createRSGroupTable, it will encounter
Exception as follows,

 
{noformat}
Exception in thread "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
java.lang.IllegalArgumentException
at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
while ProcedureExecutor  initialized by calling
{code:java}
848  startServiceThreads();{code}
And LoadBalancer is initialized by calling 
{code:java}
868  this.balancer.initialize();{code}
When LoadBalancer hasn't been initialized before createRSGroupTable, it will encounters
Exception as follows,

2018-02-02,16:12:45,688 ERROR org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG:
Uncaught runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure
table=hbase:rsgroup
 java.lang.NullPointerException
 at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
 at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
 at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
 at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
 at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
 at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
 at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)

 

As a result of CreateTableProcedure.rollbackState, it may then print logs warning TableExistsException
as follows,

2018-02-02,16:12:55,503 WARN org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
Failed to perform check
 java.io.IOException: Failed to create group table. org.apache.hadoop.hbase.TableExistsException:
hbase:rsgroup
 at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)

 

After some auto-retries, it loops running the thread RSGroupStartupWorker, will print logs
as follows, 

2018-02-02,16:23:17,626 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
 2018-02-02,16:23:17,730 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
 2018-02-02,16:23:17,834 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
 2018-02-02,16:23:17,937 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true

 

And using shells of rsgroup, it will tips that currently is in "offline mode".

 

The reason of this problem is that the order of createRSGroupTable and initializing of ProcedureExecutor
and LoadBalancer is out of control. If the former is excuted earlier, it will encounter
Exception mentioned before.

 


> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

> ---------------------------------------------------------------------------------------------
>
>                 Key: HBASE-19937
>                 URL: https://issues.apache.org/jira/browse/HBASE-19937
>             Project: HBase
>          Issue Type: Bug
>          Components: rsgroup
>    Affects Versions: 2.0.0-beta-2
>            Reporter: Xiaolin Ha
>            Assignee: Xiaolin Ha
>            Priority: Major
>         Attachments: HBASE-19937.branch-2.001.patch, HBASE-19937.branch-2.002.patch
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master load system
coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it will encounter
Exception as follows,
>  
> {noformat}
> Exception in thread "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
java.lang.IllegalArgumentException
> at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will encounters
Exception as follows,
> 2018-02-02,16:12:45,688 ERROR org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG:
Uncaught runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure
table=hbase:rsgroup
>  java.lang.NullPointerException
>  at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs warning TableExistsException
as follows,
> 2018-02-02,16:12:55,503 WARN org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
Failed to perform check
>  java.io.IOException: Failed to create group table. org.apache.hadoop.hbase.TableExistsException:
hbase:rsgroup
>  at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, will print
logs as follows, 
> 2018-02-02,16:23:17,626 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
>  2018-02-02,16:23:17,834 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
>  2018-02-02,16:23:17,937 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
>  
> And using shells of rsgroup, it will tips that currently is in "offline mode".
>  
> The reason of this problem is that the order of createRSGroupTable and initializing
of ProcedureExecutor and LoadBalancer is out of control. If the former is excuted earlier, it
will encounter Exception mentioned before.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message