hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jinglun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14591) NameNode should move the replicas to the correct storages after the storage policy is changed.
Date Fri, 21 Jun 2019 10:35:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869380#comment-16869380
] 

Jinglun commented on HDFS-14591:
--------------------------------

Hi [~jojochuang] and [~ayushtkn], thanks for your references, those are very good works and
really inspired. Actually my external service is very like the SPS. The external service automatically
downloads HDFS's image file and analyzes the image to find out the cold files, then it set
xattrs to these INodes. Another service find the INodes with the special xattrs and start
movers to move them. If I use the SPS, the progress would be: find cold files, change the
policies of them, let external-SPS handle the files. I would wait SPS be stable and consider
changing to it.
SSM is very ambitious and it handles all the replica placement jobs like balance, mover and
storage policy mismatches. It would simplify my maintenance work. Looking forward to play
with it.
I'm considering start a new type of NameNode that only handles the replica placement. It loads
images and edits and receives block reports, there won't be rpcs to the active so it won't
be a burden to my heavy NameNode. It's job is to scan all blocks and datanodes to find the
mismatched blocks and over-used dns, then move replicas to resolve the problems. Detecting
cold files could also be added to it.
Since there already has issues including storage mismatch problem, I'll close this issue.
Again thanks Wei-Chiu and Ayush for your kindly comments.:)

> NameNode should move the replicas to the correct storages after the storage policy is
changed.
> ----------------------------------------------------------------------------------------------
>
>                 Key: HDFS-14591
>                 URL: https://issues.apache.org/jira/browse/HDFS-14591
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Jinglun
>            Assignee: Jinglun
>            Priority: Major
>
> Our Xiaomi HDFS has a cluster storaging both HOT and COLD data. We have a backgroud process
searching all the files to find those that are not accessed for a period of time. Then we
set them to COLD and start a mover to move the replicas. After moving, all the replicas are
consistent with the storage policy.
> It's a natural idea to let the NameNode handle the move.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message