hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3831) Failure to renew tokens due to test-sources left in classpath
Date Tue, 21 Aug 2012 21:40:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439077#comment-13439077
] 

Jason Lowe commented on HDFS-3831:
----------------------------------

Recently we've hit a sporadic issue where the presence of the HDFS test-sources.jar prevents
a secure cluster from running properly.  Here's an example exception from a client trying
to launch a sample MapReduce job:

{noformat}
java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.submitApplication(ClientRMProtocolPBClientImpl.java:174)
	at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:318)
	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:292)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:383)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1216)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1213)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1213)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1234)
	at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:283)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
	at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:294)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
	at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
	at org.apache.hadoop.tools.TestDelegationTokenFetcher$FakeRenewer.handleKind(TestDelegationTokenFetcher.java:70)
	at org.apache.hadoop.security.token.Token.getRenewer(Token.java:286)
	at org.apache.hadoop.security.token.Token.isManaged(Token.java:301)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:279)
	at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:280)
	at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.handle(RMAppManager.java:333)
	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:255)
	at org.apache.hadoop.yarn.api.impl.pb.service.ClientRMProtocolPBServiceImpl.submitApplication(ClientRMProtocolPBServiceImpl.java:141)
	at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.callBlockingMethod(ClientRMProtocol.java:178)
	at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:353)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1528)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1524)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1522)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
	... 16 more

	at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
	at $Proxy6.submitApplication(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.submitApplication(ClientRMProtocolPBClientImpl.java:167)
	... 25 more
Caused by: java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
	at org.apache.hadoop.tools.TestDelegationTokenFetcher$FakeRenewer.handleKind(TestDelegationTokenFetcher.java:70)
	at org.apache.hadoop.security.token.Token.getRenewer(Token.java:286)
	at org.apache.hadoop.security.token.Token.isManaged(Token.java:301)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:279)
	at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:280)
	at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.handle(RMAppManager.java:333)
	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:255)
	at org.apache.hadoop.yarn.api.impl.pb.service.ClientRMProtocolPBServiceImpl.submitApplication(ClientRMProtocolPBServiceImpl.java:141)
	at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.callBlockingMethod(ClientRMProtocol.java:178)
	at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:353)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1528)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1524)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1522)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
	... 16 more

	at org.apache.hadoop.ipc.Client.call(Client.java:1088)
	at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
	... 27 more
{noformat}

What appears to be happening is that the service loader is finding {{TestDelegationTokenFetcher$FakeRenewer}}
as a token renewer and attempts to instantiate and call its {{handleKind}} to see if it can
handle renewing an HDFS token.  However loading this class causes it to try to load Mockito
which isn't available and the submission is aborted.

So why is it even finding {{TestDelegationTokenFetcher$FakeRenewer}}?  The HDFS tests-sources.jar
provides a file that registers it with the service loader.  If that jar is in the classpath
then we could attempt to load it if we don't find a suitable token renewer earlier in the
classpath.  In the dist tarball, the HDFS test-sources.jar is located in the same directory
as the main HDFS jar, and hadoop-config.sh sets up the classpath via globbing of that directory.
 Globbing in a classpath does not guarantee ordering of jars within the classpath, so sometimes
the tests-sources.jar is appearing before the main HDFS jar and that breaks the cluster when
it comes to renewing HDFS tokens.

Even if the ordering were fixed, this could end up breaking other kinds of tokens like Hive
if those token renewers appear later in the classpath than HDFS.  Seems like we should reorganize
the dist tarball so that test and test-sources jars are located in another directory that
we can easily filter out of the classpath in hadoop-config.sh.  There's still the issue of
semi-random ordering of the classpath when we use globbing, but that's probably another JIRA.
                
> Failure to renew tokens due to test-sources left in classpath
> -------------------------------------------------------------
>
>                 Key: HDFS-3831
>                 URL: https://issues.apache.org/jira/browse/HDFS-3831
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.23.3, 2.1.0-alpha
>            Reporter: Jason Lowe
>            Priority: Critical
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message