hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4324) JobClient can perhaps set mapreduce.job.credentials.binary rather than expect its presence?
Date Sat, 30 Jun 2012 17:40:44 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Harsh J updated MAPREDUCE-4324:
-------------------------------

    Attachment: MAPREDUCE-4324.patch

Here's a simple attempt, which modifies JobClient's initializer method alone.

With YARNRunner involved (via JobSubmitter, which ignores conf when sought), I am unsure where
to place it. I _think_ its sufficient for JobClient alone to have it, but if the code has
diverged much beyond JobClient paths when using Job (new API), then this patch may not suffice.
                
> JobClient can perhaps set mapreduce.job.credentials.binary rather than expect its presence?
> -------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4324
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4324
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv1, mrv2, security
>    Affects Versions: 0.22.0, 2.0.0-alpha
>            Reporter: Harsh J
>            Assignee: Harsh J
>         Attachments: MAPREDUCE-4324.patch
>
>
> HDFS-1007 added in this requirement property "mapreduce.job.credentials.binary", that
has lead Oozie to add the following duplicate snippet to all its Job-launching main classes
such as the Pig, Hive, MR and Sqoop actions:
> {code}
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
>             jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> {code}
> Same is required for any client program that launches a job from within a task.
> Why can't this simply be set by the JobClient initialization bits itself? If no one imagines
it causing issues, I'd like to add this snippet somewhere in JobSubmitter before it requests
NN/JT, as otherwise we'd get…
> {code}
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Delegation Token can be issued
only with kerberos or web authentication 
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5509)

> at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:536)

> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) 
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

> at java.lang.reflect.Method.invoke(Method.java:597) 
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) 
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) 
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) 
> at java.security.AccessController.doPrivileged(Native Method) 
> at javax.security.auth.Subject.doAs(Subject.java:396) 
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)

> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> at org.apache.hadoop.ipc.Client.call(Client.java:1107) 
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) 
> at $Proxy6.getDelegationToken(Unknown Source) 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

> at java.lang.reflect.Method.invoke(Method.java:597) 
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)

> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)

> at $Proxy6.getDelegationToken(Unknown Source) 
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:331) 
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:605)

> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:115)

> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)

> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:851) 
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) 
> at java.security.AccessController.doPrivileged(Native Method) 
> at javax.security.auth.Subject.doAs(Subject.java:396) 
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)

> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833) 
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807) 
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242) 
> {code}
> … or similar errors when a user submits a job from a task running in a secured cluster.
> Let me know your thoughts on this!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message