Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 21 Sep 2017 21:56:00 +0000 (UTC)
From: "Xiao Chen (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13094062.1502443813000.170760.1506030960489@Atlassian.JIRA>
In-Reply-To: <JIRA.13094062.1502443813000@Atlassian.JIRA>
References: <JIRA.13094062.1502443813000@Atlassian.JIRA> <JIRA.13094062.1502443813467@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to
 recursively iterate and satisfy storage policy of all the files under the
 given dir
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 21 Sep 2017 21:56:05 -0000


    [ https://issues.apache.org/jira/browse/HDFS-12291?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1617=
5522#comment-16175522 ]=20

Xiao Chen commented on HDFS-12291:
----------------------------------

Thank you for the ping [~umamaheswararao], thanks [~surendrasingh] and all =
for the work!

A few comments:
- {{checkINodeReady}} for SPS should still check the basics of the NN (stan=
dby, safemode), since it's running within a readlock.
- From existing code, I think we'd better assert no lock is held at the beg=
inning of {{throttle}} method.
- Not sure if we worry about create/delete inside the base dir during trave=
rsal here. This was important for re-encryption, where a bunch of race test=
s are added in TestReencryption.=20
- Heads up: there's a recent bug/improvement HDFS-12518 for HDFS-10899, wil=
l likely create some conflicts here. Issue itself is mostly for re-encrypti=
on though.

Otherwise LGTM

> [SPS]: Provide a mechanism to recursively iterate and satisfy storage pol=
icy of all the files under the given dir
> -------------------------------------------------------------------------=
----------------------------------------
>
>                 Key: HDFS-12291
>                 URL: https://issues.apache.org/jira/browse/HDFS-12291
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>            Reporter: Rakesh R
>            Assignee: Surendra Singh Lilhore
>         Attachments: HDFS-12291-HDFS-10285-01.patch, HDFS-12291-HDFS-1028=
5-02.patch, HDFS-12291-HDFS-10285-03.patch, HDFS-12291-HDFS-10285-04.patch
>
>
> For the given source path directory, presently SPS consider only the file=
s immediately under the directory(only one level of scanning) for satisfyin=
g the policy. It WON=E2=80=99T do recursive directory scanning and then sch=
edules SPS tasks to satisfy the storage policy of all the files till the le=
af node.=20
> The idea of this jira is to discuss & implement an efficient recursive di=
rectory iteration mechanism and satisfies storage policy for all the files =
under the given directory.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org