Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5973610EB0 for ; Fri, 30 May 2014 17:15:03 +0000 (UTC) Received: (qmail 53068 invoked by uid 500); 30 May 2014 17:15:03 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 53031 invoked by uid 500); 30 May 2014 17:15:03 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 53022 invoked by uid 99); 30 May 2014 17:15:03 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 May 2014 17:15:03 +0000 Date: Fri, 30 May 2014 17:15:03 +0000 (UTC) From: "Andrew Purtell (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-11275) [AccessController] postCreateTable hook fails when another CP creates table on their startup MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013956#comment-14013956 ] Andrew Purtell commented on HBASE-11275: ---------------------------------------- bq. Saw the exception trace when running TestVisibilityLabelsWithACL We should improve this test, because it passes 25/25 times on Java 6 and Java 7 on my test rigs. > [AccessController] postCreateTable hook fails when another CP creates table on their startup > -------------------------------------------------------------------------------------------- > > Key: HBASE-11275 > URL: https://issues.apache.org/jira/browse/HBASE-11275 > Project: HBase > Issue Type: Bug > Components: security > Reporter: Anoop Sam John > Assignee: Anoop Sam John > Fix For: 0.99.0, 0.98.4 > > Attachments: HBASE-11275.patch > > > I am using AC and VC together.( In the required order with AC first) > Still I am getting the below Exception > {code} > 2014-05-29 14:10:55,574 ERROR [ActiveMasterManager] client.AsyncProcess(313): Failed to get region location > org.apache.hadoop.hbase.TableNotFoundException: hbase:acl > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1056) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:995) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:952) > at org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:380) > at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:309) > at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:263) > at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1022) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957) > at org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174) > at org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867) > at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258) > at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123) > at org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231) > at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390) > at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619) > at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462) > at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258) > at java.lang.Thread.run(Thread.java:745) > 2014-05-29 14:10:55,587 ERROR [ActiveMasterManager] master.HMaster(621): Coprocessor postStartMaster() hook failed > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: hbase:acl: 1 time, > at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:180) > at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1(AsyncProcess.java:179) > at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1024) > at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1025) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957) > at org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174) > at org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867) > at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258) > at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123) > at org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231) > at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390) > at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619) > at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462) > at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258) > at java.lang.Thread.run(Thread.java:745) > {code} > During the master start, the postStartMaster on the AC is called 1st and it creates acl table. This create will be handled by another thread in master. > Then CP f/w calls postStartMaster on VC which will try creating labels table. > And in the AC we have a postCreateTable() hook where we add the table owner permission to acl table. This hook will be called before the actual creation of the table (labels) and will be quickly called. > So by this time, when this hook on AC is getting executed, the table creation threads might be in process of creating the tables. (acl and labels tables) > And so we will get a TableNotFoundException. > Well this can be generic issue. Another CPs also might create tables. So I believe we should handle this in AC. > Solution will be simple > Handling of the owner permission to be moved to postCreateTableHandler() CP. This hook will be called after the table is fully created by the handler thread. I think we should add the table related permission only after it is fully created. > Initailly I was thinking that we should add some waiting in the AC#postCreateTableHandler() so that the ACL table is created. (The table creations happen in another thread) > But seems this is not needed. > Even if the create table is given as a Threadpool.submit request, this pool is having only one Thread. So creation of the tables are happening in sequential way. > Here is the master code with one thread > {code} > // We depend on there being only one instance of this executor running > // at a time. To do concurrency, would need fencing of enable/disable of > // tables. > this.service.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS, 1); > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)