hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
Date Sat, 02 Sep 2017 04:01:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151401#comment-16151401
] 

Xiao Chen commented on HDFS-12291:
----------------------------------

Thanks for the thoughts [~umamaheswararao].

I think it makes sense to require SPS call for rename ops here. Assuming rename races are
handled correctly. E.g. 
# rename on a dir
# dir added file X to the batch, which is then being processed in the {{storageMovementNeeded}}
# rename of X -> Y happened, then SPS for Y for a different storage policy
I have not get a chance to check the code, but since step 2 tracks by inode id, I think we're
fine.

bq. Later when we enable automatic SPS, this should be handed automatically. 
Also makes sense. Now it sounds like whoever does phase 2 first will handle the rename with
pride. :)

> [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the
files under the given dir
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-12291
>                 URL: https://issues.apache.org/jira/browse/HDFS-12291
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>            Reporter: Rakesh R
>            Assignee: Surendra Singh Lilhore
>         Attachments: HDFS-12291-HDFS-10285-01.patch, HDFS-12291-HDFS-10285-02.patch
>
>
> For the given source path directory, presently SPS consider only the files immediately
under the directory(only one level of scanning) for satisfying the policy. It WON’T do recursive
directory scanning and then schedules SPS tasks to satisfy the storage policy of all the files
till the leaf node. 
> The idea of this jira is to discuss & implement an efficient recursive directory
iteration mechanism and satisfies storage policy for all the files under the given directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message