hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13697) DFSClient should instantiate and cache KMSClientProvider using UGI at creation time for consistent UGI handling
Date Mon, 06 Aug 2018 22:41:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570874#comment-16570874
] 

Xiao Chen commented on HDFS-13697:
----------------------------------

Thanks for the continued work and an offline discussion [~zvenczel].

As it turns out, this part may be better discussed with code than anything else. :)  I updated
a [^HDFS-13697.prelim.patch]  which kinda shows what I'm getting at. (Only verified it can
pass {{TestKMS}} and {{TestSecureEncryptionZoneWithKMS}})  The meta point is, we should no
longer morph the UGI dynamically at method calls. {{testProxyUser}} fails with this, but with
some debugging I think it's 2-folded: the test itself should create the provider within the
doAs, otherwise it's all testing the 'clientUgi'. With the provider created, the test now
fails with or without our change. Debugging into the authenticator, it's out from the {{ProxyUsers.authorize}}
check inside {{DelegationTokenAuthenticationFilter#doFilter}}, and this becomes a test problem
(which we should still look into).

bq. I've reverted HADOOP-13749 and these lines were introduced by it. I'm not sure if it makes
sense to set the login user even after the revert. What do you think?
I think we should keep the change and not revert those lines, to make sure the test case HADOOP-13749
wanted to add still passes.

bq. KeyProviderSupplier
As clarified offline, my only concern is that currently we have {{keyProvider}} and {{keyProviderSuplier}}
variables in DFSClient class, which is a code smell. Some mechanisms to make sure we only
have one would be good.

> DFSClient should instantiate and cache KMSClientProvider using UGI at creation time for
consistent UGI handling
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-13697
>                 URL: https://issues.apache.org/jira/browse/HDFS-13697
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Zsolt Venczel
>            Assignee: Zsolt Venczel
>            Priority: Major
>         Attachments: HDFS-13697.01.patch, HDFS-13697.02.patch, HDFS-13697.03.patch, HDFS-13697.04.patch,
HDFS-13697.05.patch, HDFS-13697.06.patch, HDFS-13697.07.patch, HDFS-13697.prelim.patch
>
>
> While calling KeyProviderCryptoExtension decryptEncryptedKey the call stack might not
have doAs privileged execution call (in the DFSClient for example). This results in loosing
the proxy user from UGI as UGI.getCurrentUser finds no AccessControllerContext and does a
re-login for the login user only.
> This can cause the following for example: if we have set up the oozie user to be entitled
to perform actions on behalf of example_user but oozie is forbidden to decrypt any EDEK (for
security reasons), due to the above issue, example_user entitlements are lost from UGI and
the following error is reported:
> {code}
> [0] 
> SERVER[xxx] USER[example_user] GROUP[-] TOKEN[] APP[Test_EAR] JOB[0020905-180313191552532-oozie-oozi-W]
ACTION[0020905-180313191552532-oozie-oozi-W@polling_dir_path] Error starting action [polling_dir_path].
ErrorType [ERROR], ErrorCode [FS014], Message [FS014: User [oozie] is not authorized to perform
[DECRYPT_EEK] on key with ACL name [encrypted_key]!!]
> org.apache.oozie.action.ActionExecutorException: FS014: User [oozie] is not authorized
to perform [DECRYPT_EEK] on key with ACL name [encrypted_key]!!
>  at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
>  at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:441)
>  at org.apache.oozie.action.hadoop.FsActionExecutor.touchz(FsActionExecutor.java:523)
>  at org.apache.oozie.action.hadoop.FsActionExecutor.doOperations(FsActionExecutor.java:199)
>  at org.apache.oozie.action.hadoop.FsActionExecutor.start(FsActionExecutor.java:563)
>  at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
>  at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
>  at org.apache.oozie.command.XCommand.call(XCommand.java:286)
>  at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
>  at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>  at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User [oozie]
is not authorized to perform [DECRYPT_EEK] on key with ACL name [encrypted_key]!!
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>  at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
>  at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:607)
>  at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:565)
>  at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:832)
>  at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:209)
>  at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:205)
>  at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
>  at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:205)
>  at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
>  at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1440)
>  at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1542)
>  at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1527)
>  at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:408)
>  at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:401)
>  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:401)
>  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:344)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:923)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:904)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:801)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:790)
>  at org.apache.oozie.action.hadoop.FsActionExecutor.touchz(FsActionExecutor.java:519){code}
> The operation should have succeeded as [example_user] is the owner of the [encrypted_key]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message