hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From brando...@apache.org
Subject hadoop git commit: HDFS-7976. Update NFS user guide for mount option 'sync' to minimize or avoid reordered writes. Contributed by Brandon Li
Date Tue, 24 Mar 2015 17:29:39 GMT
Repository: hadoop
Updated Branches:
  refs/heads/branch-2 e39c58fd9 -> 95bde8898


HDFS-7976. Update NFS user guide for mount option 'sync' to minimize or avoid reordered writes.
Contributed by Brandon Li

(cherry picked from commit 51f1f4937402c63392854443ccde18c1579e3348)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/95bde889
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/95bde889
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/95bde889

Branch: refs/heads/branch-2
Commit: 95bde8898636156a442949535fd776ccd638e337
Parents: e39c58f
Author: Brandon Li <brandonli@apache.org>
Authored: Tue Mar 24 10:28:38 2015 -0700
Committer: Brandon Li <brandonli@apache.org>
Committed: Tue Mar 24 10:29:27 2015 -0700

----------------------------------------------------------------------
 hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt               |  3 +++
 .../hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md       | 10 +++++++---
 2 files changed, 10 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/95bde889/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index dda75f9..5f289dd 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -479,6 +479,9 @@ Release 2.7.0 - UNRELEASED
     HDFS-7956. Improve logging for DatanodeRegistration.
     (Plamen Jeliazkov via shv)
 
+    HDFS-7976. Update NFS user guide for mount option "sync" to minimize or
+    avoid reordered writes. (brandonli)
+
   OPTIMIZATIONS
 
     HDFS-7454. Reduce memory footprint for AclEntries in NameNode.

http://git-wip-us.apache.org/repos/asf/hadoop/blob/95bde889/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md
index f1bd696..f6e4a69 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md
@@ -126,7 +126,8 @@ It's strongly recommended for the users to update a few configuration
properties
         </property>
 
 *   Users are expected to update the file dump directory. NFS client often
-    reorders writes. Sequential writes can arrive at the NFS gateway at random
+    reorders writes, especially when the export is not mounted with "sync" option.
+    Sequential writes can arrive at the NFS gateway at random
     order. This directory is used to temporarily save out-of-order writes
     before writing to HDFS. For each file, the out-of-order writes are dumped after
     they are accumulated to exceed certain threshold (e.g., 1MB) in memory.
@@ -254,13 +255,16 @@ Verify validity of NFS related services
 Mount the export "/"
 --------------------
 
-Currently NFS v3 only uses TCP as the transportation protocol. NLM is not supported so mount
option "nolock" is needed. It's recommended to use hard mount. This is because, even after
the client sends all data to NFS gateway, it may take NFS gateway some extra time to transfer
data to HDFS when writes were reorderd by NFS client Kernel.
+Currently NFS v3 only uses TCP as the transportation protocol. NLM is not supported so mount
option "nolock" is needed. 
+Mount option "sync" is strongly recommended since it can minimize or avoid reordered writes,
which results in more predictable throughput.
+ Not specifying the sync option may cause unreliable behavior when uploading large files.
+ It's recommended to use hard mount. This is because, even after the client sends all data
to NFS gateway, it may take NFS gateway some extra time to transfer data to HDFS when writes
were reorderd by NFS client Kernel.
 
 If soft mount has to be used, the user should give it a relatively long timeout (at least
no less than the default timeout on the host) .
 
 The users can mount the HDFS namespace as shown below:
 
-     [root]>mount -t nfs -o vers=3,proto=tcp,nolock,noacl $server:/  $mount_point
+     [root]>mount -t nfs -o vers=3,proto=tcp,nolock,noacl,sync $server:/  $mount_point
 
 Then the users can access HDFS as part of the local file system except that, hard link and
random write are not supported yet. To optimize the performance of large file I/O, one can
increase the NFS transfer size(rsize and wsize) during mount. By default, NFS gateway supports
1MB as the maximum transfer size. For larger data transfer size, one needs to update "nfs.rtmax"
and "nfs.rtmax" in hdfs-site.xml.
 


Mime
View raw message