hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-10285) Storage Policy Satisfier in Namenode
Date Sun, 18 Feb 2018 19:38:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368648#comment-16368648
] 

Rakesh R edited comment on HDFS-10285 at 2/18/18 7:37 PM:
----------------------------------------------------------

We have worked on the comments and following is quick update. 

{{Comment-1) }} => DONE via HDFS-13097
 {{Comment-2) }} => DONE via HDFS-13097
 {{Comment-5)}}   => DONE via HDFS-13097
 {{Comment-8) }} => DONE via HDFS-13097
 {{Comment-10)}} => DONE via HDFS-13110
 {{Comment-11)}} => DONE via HDFS-13097
 {{Comment-12)}} => DONE via HDFS-13097
 {{Comment-13)}} => DONE via HDFS-13110
 {{Comment-15)}} => DONE via HDFS-13097
 {{Comment-16)}} => DONE via HDFS-13097
 {{Comment-18)}} => DONE via HDFS-13097
 {{Comment-19)}} => DONE via HDFS-13097
 {{Comment-22)}} => DONE via HDFS-13097

 *For the below comments*, it would be great to hear your thoughts. Please let me know your
feedback on my reply.
 {{Comment-3)}} => This comment has two parts ibr and data transfer. IBR will be explored
and implemented via HDFS-13165 sub-task. But, data transfer part is not concluded yet. How
do we incorporate local move into this, currently data transfer is not having such logic.
IIUC, DNA_TRANSFER is used to send a copy of a block to another datanode. Also, mover tool
uses replaceBlock() for block movement, which already has block movement to a different storage
within the same datanode. How abt using {{replaceBlock}} pattern here in sps as well?
 {{Comment-4)}} => Depends on comment-3
 {{Comment-6, Comment-9, Comment-14, Comment-17)}} => Needs to understand more on this.
 {{Comment-20}} => Depends on comment-3

*In Progress tasks:*
 {{Comment-3)}} => HDFS-13165, this jira will only implement logic to collects back the
moved block via IBR.
 {{Comment-21)}} => HDFS-13165
 {{Comment-7)}} => HDFS-13166


was (Author: rakeshr):
We have worked on the comments and following is quick update. 

{{Comment-1)}} => DONE via HDFS-1309
 {{Comment-2)}} => DONE via HDFS-13097
 {{Comment-5)}} => DONE via HDFS-13097
 {{Comment-8)}} => DONE via HDFS-13097
 {{Comment-10)}} => DONE via HDFS-13110
 {{Comment-11)}} => DONE via HDFS-13097
 {{Comment-12)}} => DONE via HDFS-13097
 {{Comment-13)}} => DONE via HDFS-13110
 {{Comment-15)}} => DONE via HDFS-13097
 {{Comment-16)}} => DONE via HDFS-13097
 {{Comment-18)}} => DONE via HDFS-13097
 {{Comment-19)}} => DONE via HDFS-13097
 {{Comment-22)}} => DONE via HDFS-13097

 *For the below comments*, it would be great to hear your thoughts. Please let me know your
feedback on my reply.
 {{Comment-3)}} => This comment has two parts ibr and data transfer. IBR will be explored
and implemented via HDFS-13165 sub-task. But, data transfer part is not concluded yet. How
do we incorporate local move into this, currently data transfer is not having such logic.
IIUC, DNA_TRANSFER is used to send a copy of a block to another datanode. Also, mover tool
uses replaceBlock() for block movement, which already has block movement to a different storage
within the same datanode. How abt using \{{replaceBlock}} pattern here in sps as well?
 {{Comment-4}} => Depends on comment-3
 {{Comment-6, Comment-9, Comment-14, Comment-17}} => Needs to understand more on this.
 {{Comment-20}} => Depends on comment-3

*In Progress tasks:*
 {{Comment-3)}} => HDFS-13165, this jira will only implement logic to collects back the
moved block via IBR.
 {{Comment-21)}} => HDFS-13165
 {{Comment-7)}} => HDFS-13166

> Storage Policy Satisfier in Namenode
> ------------------------------------
>
>                 Key: HDFS-10285
>                 URL: https://issues.apache.org/jira/browse/HDFS-10285
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode, namenode
>    Affects Versions: HDFS-10285
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Major
>         Attachments: HDFS-10285-consolidated-merge-patch-00.patch, HDFS-10285-consolidated-merge-patch-01.patch,
HDFS-10285-consolidated-merge-patch-02.patch, HDFS-10285-consolidated-merge-patch-03.patch,
HDFS-10285-consolidated-merge-patch-04.patch, HDFS-10285-consolidated-merge-patch-05.patch,
HDFS-SPS-TestReport-20170708.pdf, SPS Modularization.pdf, Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf,
Storage-Policy-Satisfier-in-HDFS-May10.pdf, Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf
>
>
> Heterogeneous storage in HDFS introduced the concept of storage policy. These policies
can be set on directory/file to specify the user preference, where to store the physical block.
When user set the storage policy before writing data, then the blocks could take advantage
of storage policy preferences and stores physical block accordingly. 
> If user set the storage policy after writing and completing the file, then the blocks
would have been written with default storage policy (nothing but DISK). User has to run the
‘Mover tool’ explicitly by specifying all such file names as a list. In some distributed
system scenarios (ex: HBase) it would be difficult to collect all the files and run the tool
as different nodes can write files separately and file can have different paths.
> Another scenarios is, when user rename the files from one effected storage policy file
(inherited policy from parent directory) to another storage policy effected directory, it
will not copy inherited storage policy from source. So it will take effect from destination
file/dir parent storage policy. This rename operation is just a metadata change in Namenode.
The physical blocks still remain with source storage policy.
> So, Tracking all such business logic based file names could be difficult for admins from
distributed nodes(ex: region servers) and running the Mover tool. 
> Here the proposal is to provide an API from Namenode itself for trigger the storage policy
satisfaction. A Daemon thread inside Namenode should track such calls and process to DN as
movement commands. 
> Will post the detailed design thoughts document soon. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message