hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
Date Thu, 21 Sep 2017 21:56:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175522#comment-16175522
] 

Xiao Chen commented on HDFS-12291:
----------------------------------

Thank you for the ping [~umamaheswararao], thanks [~surendrasingh] and all for the work!

A few comments:
- {{checkINodeReady}} for SPS should still check the basics of the NN (standby, safemode),
since it's running within a readlock.
- From existing code, I think we'd better assert no lock is held at the beginning of {{throttle}}
method.
- Not sure if we worry about create/delete inside the base dir during traversal here. This
was important for re-encryption, where a bunch of race tests are added in TestReencryption.

- Heads up: there's a recent bug/improvement HDFS-12518 for HDFS-10899, will likely create
some conflicts here. Issue itself is mostly for re-encryption though.

Otherwise LGTM

> [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the
files under the given dir
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-12291
>                 URL: https://issues.apache.org/jira/browse/HDFS-12291
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>            Reporter: Rakesh R
>            Assignee: Surendra Singh Lilhore
>         Attachments: HDFS-12291-HDFS-10285-01.patch, HDFS-12291-HDFS-10285-02.patch,
HDFS-12291-HDFS-10285-03.patch, HDFS-12291-HDFS-10285-04.patch
>
>
> For the given source path directory, presently SPS consider only the files immediately
under the directory(only one level of scanning) for satisfying the policy. It WON’T do recursive
directory scanning and then schedules SPS tasks to satisfy the storage policy of all the files
till the leaf node. 
> The idea of this jira is to discuss & implement an efficient recursive directory
iteration mechanism and satisfies storage policy for all the files under the given directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message