From common-issues-return-31561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri May 18 01:30:33 2012 Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7145C9352 for ; Fri, 18 May 2012 01:30:33 +0000 (UTC) Received: (qmail 7892 invoked by uid 500); 18 May 2012 01:30:33 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 7845 invoked by uid 500); 18 May 2012 01:30:33 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 7837 invoked by uid 99); 18 May 2012 01:30:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 May 2012 01:30:33 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 May 2012 01:30:30 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id CFD828458 for ; Fri, 18 May 2012 01:30:08 +0000 (UTC) Date: Fri, 18 May 2012 01:30:08 +0000 (UTC) From: "Aaron T. Myers (JIRA)" To: common-issues@hadoop.apache.org Message-ID: <2123246699.12113.1337304608852.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <2015336447.12105.1337304247036.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Updated] (HADOOP-8408) MR doesn't work with a non-default ViewFS mount table and security enabled MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8408: ----------------------------------- Attachment: HDFS-8408.patch Here's a patch which addresses the issue. The trouble is that ViewFileSystem didn't override FileSystem#getCanonicalServiceName, and thus used the default which assumes that the host name of the URI can be turned into an IP address. Without this patch, running an MR job on a view filesystem with a non-default mount table name yields the following exception: {noformat} java.lang.IllegalArgumentException: java.net.UnknownHostException: vfs-cluster at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:400) at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:281) at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:235) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:124) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:101) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:81) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:410) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:325) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244) at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306) at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: java.net.UnknownHostException: vfs-cluster at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:401) ... 31 more {noformat} In addition to the automated tests in this patch, I also verified manually that I can successfully run MR jobs on a secure cluster with a non-default mount table name. > MR doesn't work with a non-default ViewFS mount table and security enabled > -------------------------------------------------------------------------- > > Key: HADOOP-8408 > URL: https://issues.apache.org/jira/browse/HADOOP-8408 > Project: Hadoop Common > Issue Type: Bug > Components: fs > Affects Versions: 2.0.0 > Reporter: Aaron T. Myers > Assignee: Aaron T. Myers > Attachments: HDFS-8408.patch > > > With security enabled, if one sets up a ViewFS mount table using the default mount table name, everything works as expected. However, if you try to create a ViewFS mount table with a non-default name, you'll end up getting an error like the following (in this case "vfs-cluster" was the name of the mount table) when running an MR job: > {noformat} > java.lang.IllegalArgumentException: java.net.UnknownHostException: vfs-cluster > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira