hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Chu <allegro...@gmail.com>
Subject Re: Question about YARN security tokens
Date Wed, 30 Apr 2014 02:06:19 GMT
Thanks for the extra information!

For reference, the code that I was running was basically the code that can
be found here: https://github.com/hortonworks/simple-yarn-app. The issue
that I was having was when running from within intellij (aka without the
classpath bits that 'hadoop classpath' adds) against either a
psuedo-distributed-mode cluster or against the MiniYARNCluster. Running
applications with the correct hadoop classpath seems to have made any
SecretManager errors go away.

Robert

Robert Chu


On Fri, Apr 25, 2014 at 4:53 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:

> Without looking at the code its hard to say. Perhaps looking at a working
> code will put you in the right direction.
> For example, here is the DistributedShell from Hadoop (only few classes)
> https://github.com/apache/hadoop-common/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell
> That is what I used a a model to dig into some of the YARN internals as I
> am playing around with ideas on how to simplify YARN API
> https://github.com/olegz/yaya, so you can use that code as well to see if
> you can get some more information from it.
>
> Cheers
> Oleg
>
>
> On Fri, Apr 25, 2014 at 12:22 AM, Robert Chu <allegrormc@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I'm new to YARN and was trying to write a simple YARN application that
>> starts up an ApplicationMaster that starts up an in-process Jetty server
>> just a simple test (this application requests no further resource
>> containers). An attempt is made to register as an application with the
>> ResourceManager while starting the ApplicationMaster process but this fails
>> with an exception related to security tokens when using the MiniYARNCluster:
>>
>> Exception in thread "main"
>> org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
>> found for ApplicationAttempt appattempt_1398290928870_0001_000002
>>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>  at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>  at
>> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
>> at
>> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
>>  at
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>  at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>  at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>>  at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source)
>>  at
>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196)
>> at
>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138)
>>  at
>> org.kiji.scoring.server.yarn.YarnServiceMaster.start(YarnServiceMaster.java:228)
>> at
>> org.kiji.scoring.server.yarn.YarnServiceMaster.main(YarnServiceMaster.java:333)
>> Caused by:
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
>> Password not found for ApplicationAttempt
>> appattempt_1398290928870_0001_000002
>> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
>>  at org.apache.hadoop.ipc.Client.call(Client.java:1362)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>  at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
>>  ... 11 more
>>
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
>> at org.apache.hadoop.util.Shell.run(Shell.java:418)
>>  at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> Upon getting this stack trace, I attempted to find
>> documentation/information/examples about security tokens but was unable to
>> find anything of note.
>>
>> Am I doing something dumb here and is this not related to security
>> tokens? If not, is there documentation or good examples around how to use
>> security tokens while launching an application?
>>
>> Thanks!
>>
>> Robert Chu
>>
>
>

Mime
View raw message