hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4451) fairscheduler fail to init job with kerberos authentication configured
Date Mon, 27 Aug 2012 21:03:10 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442717#comment-13442717
] 

Alejandro Abdelnur commented on MAPREDUCE-4451:
-----------------------------------------------

Why not using a ThreadPoolExecutor? I.e.:

{code}
        BlockingQueue<Runnable> queue = new ArrayBlockingQueue<Runnable>(MAX_QUEUE_SIZE);
        executor = new ThreadPoolExecutor(MAX_THEADS, MAX_THREADS, 10, TimeUnit.SECONDS, queue);
{code}

This will get rid of the JobInitMananager innerclass and the notify() calls. in the JobInitializer.

Just be aware of the following trick you'll have to do together with the initialization of
the executor.

{code}
        // IMPORTANT: The ThreadPoolExecutor does not always the execute
        // commands out of the queue, there are
        // certain conditions where commands are pushed directly to a thread.
        // As we are using a queue with DELAYED semantics (i.e. execute the
        // command in 5 mins) we need to make
        // sure that the commands are always pushed to the queue.
        // To achieve this (by looking a the ThreadPoolExecutor.execute()
        // implementation, we are making the pool
        // minimum size equals to the maximum size (thus threads are keep always
        // running) and we are warming up
        // all those threads (the for loop that runs dummy runnables).
        for (int i = 0; i < threads; i++) {
            executor.execute(new Runnable() {
                public void run() {
                    try {
                        Thread.sleep(100);
                    }
                    catch (InterruptedException ex) {
                        log.warn("Could not warm up threadpool {0}", ex.getMessage(), ex);
                    }
                }
            });
        }
{code}
                
> fairscheduler fail to init job with kerberos authentication configured
> ----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4451
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4451
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 1.0.3
>            Reporter: Erik.fang
>             Fix For: 1.1.0
>
>         Attachments: MAPREDUCE-4451_branch-1.patch, MAPREDUCE-4451_branch-1.patch, MAPREDUCE-4451_branch-1.patch
>
>
> Using FairScheduler in Hadoop 1.0.3 with kerberos authentication configured. Job initialization
fails:
> {code}
> 2012-07-17 15:15:09,220 ERROR org.apache.hadoop.mapred.JobTracker: Job initialization
failed:
> java.io.IOException: Call to /192.168.7.80:8020 failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials
provided (Mechanism level: Failed to find any Kerberos tgt)]
>         at org.apache.hadoop.ipc.Client.wrapException(Client.java:1129)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1097)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>         at $Proxy7.getProtocolVersion(Unknown Source)
>         at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:411)
>         at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:125)
>         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:329)
>         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:294)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
>         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1411)
>         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1429)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
>         at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
>         at org.apache.hadoop.security.Credentials.writeTokenStorageFile(Credentials.java:169)
>         at org.apache.hadoop.mapred.JobInProgress.generateAndStoreTokens(JobInProgress.java:3558)
>         at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:696)
>         at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3911)
>         at org.apache.hadoop.mapred.FairScheduler$JobInitializer$InitJob.run(FairScheduler.java:301)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any
Kerberos tgt)]
>         at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:543)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
>         at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:488)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:590)
>         at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:187)
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1228)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1072)
>         ... 20 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>         at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
>         at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134)
>         at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:385)
>         at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:187)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:583)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:580)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:579)
>         ... 23 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
>         at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
>         at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
>         at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
>         at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
>         at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
>         ... 32 more
> {code}
> When a job is submitted, fairscheduler calls JobTracker.initJob, which calls JobInProgress.generateAndStoreTokens
to write security keys to hdfs. However, the operation is involved in the server side rpc
call path, using UGI created by UserGroupInformation.createRemoteUser in rpc server, which
have no tgt. This should be done with UGI used by JobTracker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message