impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mostafa Mokhtar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IMPALA-5420) Check If HDFS ACLs are enabled before trying to get the ACLs
Date Fri, 02 Jun 2017 00:43:04 GMT
Mostafa Mokhtar created IMPALA-5420:
---------------------------------------

             Summary: Check If HDFS ACLs are enabled before trying to get the ACLs 
                 Key: IMPALA-5420
                 URL: https://issues.apache.org/jira/browse/IMPALA-5420
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
    Affects Versions: Impala 2.9.0
            Reporter: Mostafa Mokhtar


While Impala is loading block information a lot of HDFS exceptions are printed to the Catalog
log, Impala swallows those exceptions and metadata loading completes fine. 

Yet these exceptions slow down metadata loading significantly and cause quiet a bit of memory
churn.

There's an ACL bit that Impala can query on the FsPermission of the FileStatus to see if there
is an ACL worth fetching:

https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java#L303

{code}
0531 23:09:45.532622 92906 RetryInvocationHandler.java:126] Exception while invoking class
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus over null.
Not retrying because try once and fail.
Java exception follows:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AclException): The ACL
operation has been rejected.  Support for ACLs has been disabled by setting dfs.namenode.acls.enabled
to false.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAclsConfigFlag(FSNamesystem.java:9637)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAclStatus(FSNamesystem.java:9111)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAclStatus(NameNodeRpcServer.java:1620)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getAclStatus(AuthorizationProviderProxyClientProtocol.java:907)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAclStatus(ClientNamenodeProtocolServerSideTranslatorPB.java:1335)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)

	at org.apache.hadoop.ipc.Client.call(Client.java:1502)
	at org.apache.hadoop.ipc.Client.call(Client.java:1439)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
	at com.sun.proxy.$Proxy17.getAclStatus(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus(ClientNamenodeProtocolTranslatorPB.java:1336)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy18.getAclStatus(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getAclStatus(DFSClient.java:3353)
	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2059)
	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2056)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getAclStatus(DistributedFileSystem.java:2056)
	at org.apache.impala.util.FsPermissionChecker.getPermissions(FsPermissionChecker.java:288)
	at org.apache.impala.catalog.HdfsTable.getAvailableAccessLevel(HdfsTable.java:787)
	at org.apache.impala.catalog.HdfsTable.createPartition(HdfsTable.java:861)
	at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:639)
	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1065)
	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1002)
	at org.apache.impala.catalog.TableLoader.load(TableLoader.java:80)
	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:237)
	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:234)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
W0531 23:09:45.538826 92906 RetryInvocationHandler.java:126] Exception while invoking class
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus over null.
Not retrying because try once and fail.
Java exception follows:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AclException): The ACL
operation has been rejected.  Support for ACLs has been disabled by setting dfs.namenode.acls.enabled
to false.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAclsConfigFlag(FSNamesystem.java:9637)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAclStatus(FSNamesystem.java:9111)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAclStatus(NameNodeRpcServer.java:1620)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getAclStatus(AuthorizationProviderProxyClientProtocol.java:907)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAclStatus(ClientNamenodeProtocolServerSideTranslatorPB.java:1335)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)

	at org.apache.hadoop.ipc.Client.call(Client.java:1502)
	at org.apache.hadoop.ipc.Client.call(Client.java:1439)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
	at com.sun.proxy.$Proxy17.getAclStatus(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus(ClientNamenodeProtocolTranslatorPB.java:1336)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy18.getAclStatus(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getAclStatus(DFSClient.java:3353)
	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2059)
	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2056)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getAclStatus(DistributedFileSystem.java:2056)
	at org.apache.impala.util.FsPermissionChecker.getPermissions(FsPermissionChecker.java:288)
	at org.apache.impala.catalog.HdfsTable.getAvailableAccessLevel(HdfsTable.java:787)
	at org.apache.impala.catalog.HdfsTable.createPartition(HdfsTable.java:861)
	at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:639)
	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1065)
	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1002)
	at org.apache.impala.catalog.TableLoader.load(TableLoader.java:80)
	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:237)
	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:234)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
{code}





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message