Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 8 Jul 2017 17:44:00 +0000 (UTC)
From: "Surendra Singh Lilhore (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12958700.1460600514000.200403.1499535840238@Atlassian.JIRA>
In-Reply-To: <JIRA.12958700.1460600514000@Atlassian.JIRA>
References: <JIRA.12958700.1460600514000@Atlassian.JIRA> <JIRA.12958700.1460600514798@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HDFS-10285) Storage Policy Satisfier in Namenode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Sat, 08 Jul 2017 17:44:05 -0000


     [ https://issues.apache.org/jira/browse/HDFS-10285?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:all-tabpanel ]

Surendra Singh Lilhore updated HDFS-10285:
------------------------------------------
    Attachment: HDFS-SPS-TestReport-20170708.pdf

Attached test report for SPS feature. All the test cases are passing.
Please review..

> Storage Policy Satisfier in Namenode
> ------------------------------------
>
>                 Key: HDFS-10285
>                 URL: https://issues.apache.org/jira/browse/HDFS-10285
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode, namenode
>    Affects Versions: HDFS-10285
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HDFS-SPS-TestReport-20170708.pdf, Storage-Policy-Sat=
isfier-in-HDFS-June-20-2017.pdf, Storage-Policy-Satisfier-in-HDFS-May10.pdf
>
>
> Heterogeneous storage in HDFS introduced the concept of storage policy. T=
hese policies can be set on directory/file to specify the user preference, =
where to store the physical block. When user set the storage policy before =
writing data, then the blocks could take advantage of storage policy prefer=
ences and stores physical block accordingly.=20
> If user set the storage policy after writing and completing the file, the=
n the blocks would have been written with default storage policy (nothing b=
ut DISK). User has to run the =E2=80=98Mover tool=E2=80=99 explicitly by sp=
ecifying all such file names as a list. In some distributed system scenario=
s (ex: HBase) it would be difficult to collect all the files and run the to=
ol as different nodes can write files separately and file can have differen=
t paths.
> Another scenarios is, when user rename the files from one effected storag=
e policy file (inherited policy from parent directory) to another storage p=
olicy effected directory, it will not copy inherited storage policy from so=
urce. So it will take effect from destination file/dir parent storage polic=
y. This rename operation is just a metadata change in Namenode. The physica=
l blocks still remain with source storage policy.
> So, Tracking all such business logic based file names could be difficult =
for admins from distributed nodes(ex: region servers) and running the Mover=
 tool.=20
> Here the proposal is to provide an API from Namenode itself for trigger t=
he storage policy satisfaction. A Daemon thread inside Namenode should trac=
k such calls and process to DN as movement commands.=20
> Will post the detailed design thoughts document soon.=20


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org