hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-15401) ConcurrentModificationException on Subject.getPrivateCredentials in UGI constructor
Date Fri, 20 Apr 2018 17:38:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Xiao Chen updated HADOOP-15401:
-------------------------------
    Description: 
Seen a recent exception from KMS client provider as follows:
{noformat}
java.io.IOException: java.util.ConcurrentModificationException
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:488)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
        at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
        at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
        at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
        at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
Caused by: java.util.ConcurrentModificationException
        at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
        at java.util.LinkedList$ListItr.next(LinkedList.java:888)
        at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070)
        at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399)
        at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1372)
        at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767)
        at org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(KerberosUtil.java:267)
        at org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:715)
        at org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:701)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:742)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:141)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471)
        ... 12 more
{noformat}
It looks like we have ran into a race modifying jdk Subject class' privCredentials.

Found [https://bugs.openjdk.java.net/browse/JDK-4892913] but that jira was created before
Hadoop....

[~daryn], any thoughts on this?
 (We have not seen this in versions pre-3.0 yet, but it seems HADOOP-9747 would make solve
this exact exception because it removed the access in UGI constructor)

  was:
Seen a recent exception from KMS client provider as follows:

{noformat}
java.io.IOException: java.util.ConcurrentModificationException
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:488)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
        at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
        at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
        at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
        at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
Caused by: java.util.ConcurrentModificationException
        at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
        at java.util.LinkedList$ListItr.next(LinkedList.java:888)
        at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070)
        at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399)
        at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1372)
        at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767)
        at org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(KerberosUtil.java:267)
        at org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:715)
        at org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:701)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:742)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:141)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471)
        ... 12 more
{noformat}

It looks like we have ran into a race inside jdk's Subject class.

Found https://bugs.openjdk.java.net/browse/JDK-4892913 but that jira was created before Hadoop....

[~daryn], any thoughts on this?
(With all due respect, we have not seen this in versions without HADOOP-9747 yet)


> ConcurrentModificationException on Subject.getPrivateCredentials in UGI constructor
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-15401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15401
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 3.1.0, 3.0.3
>            Reporter: Xiao Chen
>            Priority: Major
>
> Seen a recent exception from KMS client provider as follows:
> {noformat}
> java.io.IOException: java.util.ConcurrentModificationException
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:488)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
>         at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
>         at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
>         at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
> Caused by: java.util.ConcurrentModificationException
>         at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
>         at java.util.LinkedList$ListItr.next(LinkedList.java:888)
>         at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070)
>         at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399)
>         at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1372)
>         at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767)
>         at org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(KerberosUtil.java:267)
>         at org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:715)
>         at org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:701)
>         at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:742)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:141)
>         at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471)
>         ... 12 more
> {noformat}
> It looks like we have ran into a race modifying jdk Subject class' privCredentials.
> Found [https://bugs.openjdk.java.net/browse/JDK-4892913] but that jira was created before
Hadoop....
> [~daryn], any thoughts on this?
>  (We have not seen this in versions pre-3.0 yet, but it seems HADOOP-9747 would make
solve this exact exception because it removed the access in UGI constructor)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message