Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3E7D110F8D for ; Fri, 6 Mar 2015 23:20:11 +0000 (UTC) Received: (qmail 43032 invoked by uid 500); 6 Mar 2015 23:20:04 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 42968 invoked by uid 500); 6 Mar 2015 23:20:04 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 42959 invoked by uid 99); 6 Mar 2015 23:20:04 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Mar 2015 23:20:04 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 7D3E2E109D; Fri, 6 Mar 2015 23:20:04 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: brandonli@apache.org To: common-commits@hadoop.apache.org Message-Id: <8ed2c3419739473592a67f5f78962971@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: hadoop git commit: HDFS-6488. Support HDFS superuser in NFSv3 gateway. Contributed by Brandon Li Date: Fri, 6 Mar 2015 23:20:04 +0000 (UTC) Repository: hadoop Updated Branches: refs/heads/trunk 27e8ea820 -> 0f8ecb1d0 HDFS-6488. Support HDFS superuser in NFSv3 gateway. Contributed by Brandon Li Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0f8ecb1d Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0f8ecb1d Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0f8ecb1d Branch: refs/heads/trunk Commit: 0f8ecb1d0ce6d3ee9a7caf5b15b299210c2b8875 Parents: 27e8ea8 Author: Brandon Li Authored: Fri Mar 6 15:19:45 2015 -0800 Committer: Brandon Li Committed: Fri Mar 6 15:19:45 2015 -0800 ---------------------------------------------------------------------- .../hadoop/hdfs/nfs/conf/NfsConfigKeys.java | 14 +++++++ .../hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java | 12 +++++- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 2 + .../src/site/markdown/HdfsNfsGateway.md | 44 +++++++++++++++++--- 4 files changed, 64 insertions(+), 8 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/0f8ecb1d/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java index 9e4aaf5..09ee579 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java +++ b/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java @@ -73,4 +73,18 @@ public class NfsConfigKeys { public static final String NFS_METRICS_PERCENTILES_INTERVALS_KEY = "nfs.metrics.percentiles.intervals"; public static final String NFS_METRICS_PERCENTILES_INTERVALS_DEFAULT = ""; + + /* + * HDFS super-user is the user with the same identity as NameNode process + * itself and the super-user can do anything in that permissions checks never + * fail for the super-user. If the following property is configured, the + * superuser on NFS client can access any file on HDFS. By default, the super + * user is not configured in the gateway. Note that, even the the superuser is + * configured, "nfs.exports.allowed.hosts" still takes effect. For example, + * the superuser will not have write access to HDFS files through the gateway + * if the NFS client host is not allowed to have write access in + * "nfs.exports.allowed.hosts". + */ + public static final String NFS_SUPERUSER_KEY = "nfs.superuser"; + public static final String NFS_SUPERUSER_DEFAULT = ""; } http://git-wip-us.apache.org/repos/asf/hadoop/blob/0f8ecb1d/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java b/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java index 05d0674..268abba 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java +++ b/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java @@ -34,7 +34,6 @@ import org.apache.hadoop.fs.CommonConfigurationKeysPublic; import org.apache.hadoop.fs.CreateFlag; import org.apache.hadoop.fs.DirectoryListingStartAfterNotFoundException; import org.apache.hadoop.fs.FSDataInputStream; -import org.apache.hadoop.fs.FileSystem.Statistics; import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.FsStatus; import org.apache.hadoop.fs.Options; @@ -166,6 +165,7 @@ public class RpcProgramNfs3 extends RpcProgram implements Nfs3Interface { private JvmPauseMonitor pauseMonitor; private Nfs3HttpServer infoServer = null; static Nfs3Metrics metrics; + private String superuser; public RpcProgramNfs3(NfsConfiguration config, DatagramSocket registrationSocket, boolean allowInsecurePorts) throws IOException { @@ -200,6 +200,9 @@ public class RpcProgramNfs3 extends RpcProgram implements Nfs3Interface { UserGroupInformation.setConfiguration(config); SecurityUtil.login(config, NfsConfigKeys.DFS_NFS_KEYTAB_FILE_KEY, NfsConfigKeys.DFS_NFS_KERBEROS_PRINCIPAL_KEY); + superuser = config.get(NfsConfigKeys.NFS_SUPERUSER_KEY, + NfsConfigKeys.NFS_SUPERUSER_DEFAULT); + LOG.info("Configured HDFS superuser is " + superuser); if (!enableDump) { writeDumpDir = null; @@ -583,13 +586,18 @@ public class RpcProgramNfs3 extends RpcProgram implements Nfs3Interface { } try { - // HDFS-5804 removed supserUserClient access attrs = writeManager.getFileAttr(dfsClient, handle, iug); if (attrs == null) { LOG.error("Can't get path for fileId: " + handle.getFileId()); return new ACCESS3Response(Nfs3Status.NFS3ERR_STALE); } + if(iug.getUserName(securityHandler.getUid(), "unknown").equals(superuser)) { + int access = Nfs3Constant.ACCESS3_LOOKUP | Nfs3Constant.ACCESS3_DELETE + | Nfs3Constant.ACCESS3_EXECUTE | Nfs3Constant.ACCESS3_EXTEND + | Nfs3Constant.ACCESS3_MODIFY | Nfs3Constant.ACCESS3_READ; + return new ACCESS3Response(Nfs3Status.NFS3_OK, attrs, access); + } int access = Nfs3Utils.getAccessRightsForUserGroup( securityHandler.getUid(), securityHandler.getGid(), securityHandler.getAuxGids(), attrs); http://git-wip-us.apache.org/repos/asf/hadoop/blob/0f8ecb1d/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index b443902..29717e1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -348,6 +348,8 @@ Release 2.7.0 - UNRELEASED HDFS-7656. Expose truncate API for HDFS httpfs. (yliu) + HDFS-6488. Support HDFS superuser in NFS gateway. (brandonli) + IMPROVEMENTS HDFS-7752. Improve description for http://git-wip-us.apache.org/repos/asf/hadoop/blob/0f8ecb1d/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md index cea491f..e6666d4 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md @@ -80,14 +80,33 @@ The above are the only required configuration for the NFS gateway in non-secure The rest of the NFS gateway configurations are optional for both secure and non-secure mode. -The AIX NFS client has a [few known issues](https://issues.apache.org/jira/browse/HDFS-6549) that prevent it from working correctly by default with the HDFS NFS Gateway. If you want to be able to access the HDFS NFS Gateway from AIX, you should set the following configuration setting to enable work-arounds for these issues: +* The AIX NFS client has a [few known issues](https://issues.apache.org/jira/browse/HDFS-6549) + that prevent it from working correctly by default with the HDFS NFS Gateway. If you want to + be able to access the HDFS NFS Gateway from AIX, you should set the following configuration + setting to enable work-arounds for these issues: - - nfs.aix.compatibility.mode.enabled - true - + + nfs.aix.compatibility.mode.enabled + true + + + Note that regular, non-AIX clients should NOT enable AIX compatibility mode. The work-arounds + implemented by AIX compatibility mode effectively disable safeguards to ensure that listing + of directory contents via NFS returns consistent results, and that all data sent to the NFS + server can be assured to have been committed. + +* HDFS super-user is the user with the same identity as NameNode process itself and + the super-user can do anything in that permissions checks never fail for the super-user. + If the following property is configured, the superuser on NFS client can access any file + on HDFS. By default, the super user is not configured in the gateway. + Note that, even the the superuser is configured, "nfs.exports.allowed.hosts" still takes effect. + For example, the superuser will not have write access to HDFS files through the gateway if + the NFS client host is not allowed to have write access in "nfs.exports.allowed.hosts". -Note that regular, non-AIX clients should NOT enable AIX compatibility mode. The work-arounds implemented by AIX compatibility mode effectively disable safeguards to ensure that listing of directory contents via NFS returns consistent results, and that all data sent to the NFS server can be assured to have been committed. + + nfs.superuser + the_name_of_hdfs_superuser + It's strongly recommended for the users to update a few configuration properties based on their use cases. All the following configuration properties can be added or updated in hdfs-site.xml. @@ -135,6 +154,19 @@ It's strongly recommended for the users to update a few configuration properties * rw +* HDFS super-user is the user with the same identity as NameNode process itself and + the super-user can do anything in that permissions checks never fail for the super-user. + If the following property is configured, the superuser on NFS client can access any file + on HDFS. By default, the super user is not configured in the gateway. + Note that, even the the superuser is configured, "nfs.exports.allowed.hosts" still takes effect. + For example, the superuser will not have write access to HDFS files through the gateway if + the NFS client host is not allowed to have write access in "nfs.exports.allowed.hosts". + + + nfs.superuser + the_name_of_hdfs_superuser + + * JVM and log settings. You can export JVM settings (e.g., heap size and GC log) in HADOOP\_NFS3\_OPTS. More NFS related settings can be found in hadoop-env.sh. To get NFS debug trace, you can edit the log4j.property file