hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aajis...@apache.org
Subject hadoop git commit: HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao.
Date Thu, 30 Apr 2015 11:15:14 GMT
Repository: hadoop
Updated Branches:
  refs/heads/branch-2.7 9828638fa -> 185a1ff27


HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir.
Contributed by Xiaoyu Yao.

(cherry picked from commit de9404f02f36bf9a1100c67f41db907d494bb9ed)
(cherry picked from commit fdb886d65642f170e65c619e49324aa208237519)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/185a1ff2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/185a1ff2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/185a1ff2

Branch: refs/heads/branch-2.7
Commit: 185a1ff275c85214e235bbcd96412398649b72a7
Parents: 9828638
Author: Akira Ajisaka <aajisaka@apache.org>
Authored: Thu Apr 30 20:12:32 2015 +0900
Committer: Akira Ajisaka <aajisaka@apache.org>
Committed: Thu Apr 30 20:13:47 2015 +0900

----------------------------------------------------------------------
 hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt                  | 3 +++
 .../hadoop-hdfs/src/main/resources/hdfs-default.xml          | 7 +++++--
 .../hadoop-hdfs/src/site/markdown/ArchivalStorage.md         | 8 ++++++++
 3 files changed, 16 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/185a1ff2/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 4ad47a4..b1c53bc 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -16,6 +16,9 @@ Release 2.7.1 - UNRELEASED
     HDFS-8204. Mover/Balancer should not schedule two replicas to the same
     datanode.  (Walter Su via szetszwo)
 
+    HDFS-7770. Need document for storage type label of data node storage
+    locations under dfs.data.dir. (Xiaoyu Yao via aajisaka)
+
   OPTIMIZATIONS
 
   BUG FIXES

http://git-wip-us.apache.org/repos/asf/hadoop/blob/185a1ff2/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
index a8c2400..43fa129 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
@@ -430,8 +430,11 @@
   <description>Determines where on the local filesystem an DFS data node
   should store its blocks.  If this is a comma-delimited
   list of directories, then data will be stored in all named
-  directories, typically on different devices.
-  Directories that do not exist are ignored.
+  directories, typically on different devices. The directories should be tagged
+  with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS
+  storage policies. The default storage type will be DISK if the directory does
+  not have a storage type tagged explicitly. Directories that do not exist will
+  be created if local filesystem permission allows.
   </description>
 </property>
 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/185a1ff2/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
index 2038401..d924f80 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
@@ -95,6 +95,14 @@ The effective storage policy can be retrieved by the "[`dfsadmin -getStoragePoli
 ### Configuration
 
 * **dfs.storage.policy.enabled** - for enabling/disabling the storage policy feature. The
default value is `true`.
+* **dfs.datanode.data.dir** - on each data node, the comma-separated storage locations should
be tagged with their storage types. This allows storage policies to place the blocks on different
storage types according to policy. For example:
+
+    1.  A datanode storage location /grid/dn/disk0 on DISK should be configured with `[DISK]file:///grid/dn/disk0`
+    2.  A datanode storage location /grid/dn/ssd0 on SSD can should configured with `[SSD]file:///grid/dn/ssd0`
+    3.  A datanode storage location /grid/dn/archive0 on ARCHIVE should be configured with
`[ARCHIVE]file:///grid/dn/archive0`
+    4.  A datanode storage location /grid/dn/ram0 on RAM_DISK should be configured with `[RAM_DISK]file:///grid/dn/ram0`
+
+    The default storage type of a datanode storage location will be DISK if it does not have
a storage type tagged explicitly.
 
 Mover - A New Data Migration Tool
 ---------------------------------


Mime
View raw message