hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11275) [AccessController] postCreateTable hook fails when another CP creates table on their startup
Date Fri, 30 May 2014 17:15:03 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013956#comment-14013956
] 

Andrew Purtell commented on HBASE-11275:
----------------------------------------

bq. Saw the exception trace when running TestVisibilityLabelsWithACL

We should improve this test, because it passes 25/25 times on Java 6 and Java 7 on my test
rigs. 

> [AccessController] postCreateTable hook fails when another CP creates table on their
startup
> --------------------------------------------------------------------------------------------
>
>                 Key: HBASE-11275
>                 URL: https://issues.apache.org/jira/browse/HBASE-11275
>             Project: HBase
>          Issue Type: Bug
>          Components: security
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 0.99.0, 0.98.4
>
>         Attachments: HBASE-11275.patch
>
>
> I am using AC and VC together.( In the required order with AC first)
> Still I am getting the below Exception
> {code}
> 2014-05-29 14:10:55,574 ERROR [ActiveMasterManager] client.AsyncProcess(313): Failed
to get region location 
> org.apache.hadoop.hbase.TableNotFoundException: hbase:acl
>        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1056)
>        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:995)
>        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:952)
>        at org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:380)
>        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:309)
>        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:263)
>        at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1022)
>        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>        at org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>        at org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>        at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>        at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>        at org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>        at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390)
>        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619)
>        at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462)
>        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258)
>        at java.lang.Thread.run(Thread.java:745)
> 2014-05-29 14:10:55,587 ERROR [ActiveMasterManager] master.HMaster(621): Coprocessor
postStartMaster() hook failed
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action:
hbase:acl: 1 time, 
>        at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:180)
>        at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1(AsyncProcess.java:179)
>        at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1024)
>        at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1025)
>        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>        at org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>        at org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>        at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>        at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>        at org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>        at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390)
>        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619)
>        at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462)
>        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258)
>        at java.lang.Thread.run(Thread.java:745)
> {code}
> During the master start, the postStartMaster on the AC is called 1st and it creates acl
table.   This create will be handled by another thread in master.
> Then CP f/w calls postStartMaster on VC which will try creating labels table.  
> And in the AC we have a postCreateTable() hook where we add the table owner permission
to acl table.  This hook will be called before the actual creation of the table (labels) and
will be quickly called.
> So by this time, when this hook on AC is getting executed, the table creation threads
might be in process of creating the tables. (acl and labels tables)
> And so we will get a TableNotFoundException.
> Well this can be generic issue. Another CPs also might create tables. So I believe we
should handle this in AC.
> Solution will be simple
> Handling of the owner permission to be moved to postCreateTableHandler() CP. This hook
will be called after the table is fully created by the handler thread. I think we should add
the table related permission only after it is fully created. 
> Initailly I was thinking that we should add some waiting in the AC#postCreateTableHandler()
so that the ACL table is created. (The table creations happen in another thread)
> But seems this is not needed.
> Even if the create table is given as a Threadpool.submit request, this pool  is having
only one Thread.  So creation of the tables are happening in sequential way. 
> Here is the master code with one thread
>    {code}
>    // We depend on there being only one instance of this executor running
>    // at a time.  To do concurrency, would need fencing of enable/disable of
>    // tables.
>    this.service.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS, 1);
>    {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message