Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EC94C17C6F for ; Thu, 30 Apr 2015 11:15:16 +0000 (UTC) Received: (qmail 71188 invoked by uid 500); 30 Apr 2015 11:15:15 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 71003 invoked by uid 500); 30 Apr 2015 11:15:15 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 70832 invoked by uid 99); 30 Apr 2015 11:15:15 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Apr 2015 11:15:14 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id CF568E0061; Thu, 30 Apr 2015 11:15:14 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: aajisaka@apache.org To: common-commits@hadoop.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: hadoop git commit: HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. Date: Thu, 30 Apr 2015 11:15:14 +0000 (UTC) Repository: hadoop Updated Branches: refs/heads/branch-2.7 9828638fa -> 185a1ff27 HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (cherry picked from commit de9404f02f36bf9a1100c67f41db907d494bb9ed) (cherry picked from commit fdb886d65642f170e65c619e49324aa208237519) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/185a1ff2 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/185a1ff2 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/185a1ff2 Branch: refs/heads/branch-2.7 Commit: 185a1ff275c85214e235bbcd96412398649b72a7 Parents: 9828638 Author: Akira Ajisaka Authored: Thu Apr 30 20:12:32 2015 +0900 Committer: Akira Ajisaka Committed: Thu Apr 30 20:13:47 2015 +0900 ---------------------------------------------------------------------- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++ .../hadoop-hdfs/src/main/resources/hdfs-default.xml | 7 +++++-- .../hadoop-hdfs/src/site/markdown/ArchivalStorage.md | 8 ++++++++ 3 files changed, 16 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/185a1ff2/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index 4ad47a4..b1c53bc 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -16,6 +16,9 @@ Release 2.7.1 - UNRELEASED HDFS-8204. Mover/Balancer should not schedule two replicas to the same datanode. (Walter Su via szetszwo) + HDFS-7770. Need document for storage type label of data node storage + locations under dfs.data.dir. (Xiaoyu Yao via aajisaka) + OPTIMIZATIONS BUG FIXES http://git-wip-us.apache.org/repos/asf/hadoop/blob/185a1ff2/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml index a8c2400..43fa129 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml @@ -430,8 +430,11 @@ Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named - directories, typically on different devices. - Directories that do not exist are ignored. + directories, typically on different devices. The directories should be tagged + with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS + storage policies. The default storage type will be DISK if the directory does + not have a storage type tagged explicitly. Directories that do not exist will + be created if local filesystem permission allows. http://git-wip-us.apache.org/repos/asf/hadoop/blob/185a1ff2/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md index 2038401..d924f80 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md @@ -95,6 +95,14 @@ The effective storage policy can be retrieved by the "[`dfsadmin -getStoragePoli ### Configuration * **dfs.storage.policy.enabled** - for enabling/disabling the storage policy feature. The default value is `true`. +* **dfs.datanode.data.dir** - on each data node, the comma-separated storage locations should be tagged with their storage types. This allows storage policies to place the blocks on different storage types according to policy. For example: + + 1. A datanode storage location /grid/dn/disk0 on DISK should be configured with `[DISK]file:///grid/dn/disk0` + 2. A datanode storage location /grid/dn/ssd0 on SSD can should configured with `[SSD]file:///grid/dn/ssd0` + 3. A datanode storage location /grid/dn/archive0 on ARCHIVE should be configured with `[ARCHIVE]file:///grid/dn/archive0` + 4. A datanode storage location /grid/dn/ram0 on RAM_DISK should be configured with `[RAM_DISK]file:///grid/dn/ram0` + + The default storage type of a datanode storage location will be DISK if it does not have a storage type tagged explicitly. Mover - A New Data Migration Tool ---------------------------------