Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 2 Sep 2017 04:01:06 +0000 (UTC)
From: "Xiao Chen (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13094062.1502443813000.14579.1504324866974@Atlassian.JIRA>
In-Reply-To: <JIRA.13094062.1502443813000@Atlassian.JIRA>
References: <JIRA.13094062.1502443813000@Atlassian.JIRA> <JIRA.13094062.1502443813467@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to
 recursively iterate and satisfy storage policy of all the files under the
 given dir
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Sat, 02 Sep 2017 04:01:14 -0000


    [ https://issues.apache.org/jira/browse/HDFS-12291?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1615=
1401#comment-16151401 ]=20

Xiao Chen commented on HDFS-12291:
----------------------------------

Thanks for the thoughts [~umamaheswararao].

I think it makes sense to require SPS call for rename ops here. Assuming re=
name races are handled correctly. E.g.=20
# rename on a dir
# dir added file X to the batch, which is then being processed in the {{sto=
rageMovementNeeded}}
# rename of X -> Y happened, then SPS for Y for a different storage policy
I have not get a chance to check the code, but since step 2 tracks by inode=
 id, I think we're fine.

bq. Later when we enable automatic SPS, this should be handed automatically=
.=20
Also makes sense. Now it sounds like whoever does phase 2 first will handle=
 the rename with pride. :)

> [SPS]: Provide a mechanism to recursively iterate and satisfy storage pol=
icy of all the files under the given dir
> -------------------------------------------------------------------------=
----------------------------------------
>
>                 Key: HDFS-12291
>                 URL: https://issues.apache.org/jira/browse/HDFS-12291
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>            Reporter: Rakesh R
>            Assignee: Surendra Singh Lilhore
>         Attachments: HDFS-12291-HDFS-10285-01.patch, HDFS-12291-HDFS-1028=
5-02.patch
>
>
> For the given source path directory, presently SPS consider only the file=
s immediately under the directory(only one level of scanning) for satisfyin=
g the policy. It WON=E2=80=99T do recursive directory scanning and then sch=
edules SPS tasks to satisfy the storage policy of all the files till the le=
af node.=20
> The idea of this jira is to discuss & implement an efficient recursive di=
rectory iteration mechanism and satisfies storage policy for all the files =
under the given directory.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org