hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7343) HDFS smart storage management
Date Fri, 24 Mar 2017 08:00:51 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939969#comment-15939969

Kai Zheng commented on HDFS-7343:

Thanks [~zhouwei] for updating the docs and addressing the questions.

[~andrew.wang] has a good asking:
bq. How often does the SSM need to poll the NN to get information? How much information each
time? Some triggers might require listing a lot of the namespace.

I had an offline discussion about this with Wei, [~HuafengWang] and [~Sammi]. To avoid listing
a lot of the namespace, one way is to let SSM server have a copy of namespace. This would
allow to evolve SSM towards really powerful and doing a lot of things without affecting NameNode
very much. So the questions now comes to:
* What kinds of file meta info are needed to make SSM function;
* How to fetch the namespace to prepare for the base;
* Given the namespace base, how to incrementally get updated time by time.

h4. What file meta info are needed
Roughly file meta info like filename, xattributes, permissions, atime/mtime, length and etc.
but not including block list info. So it's part of the namespace, very compacted, not the
whole, should be much smaller than the whole.

h4. How to fetch the namespace to prepare for the base
There're two ways for SSM to do this: 
* Fetch the fsimage and load the needed info by itself. 
* Get file status list from NameNode page by page at a NN affordable speed. There doesn't
urgent need that requires SSM to complete the work in short time. It can take some days in
a large cluster and the work can happen in idle time.

In either way, SSM can take quite much time to finish the work so a {{safe mode}} would be
needed to allow the period. During safe mode time, SSM won't allow relevant operations.

h4. Given the namespace base, how to incrementally get updated time by time
I thought both [~andrew.wang] and [~eddyxu] (by offline) suggested we can use the inotify
support. It's a good chance to exercise the function.

Note this will consider NameNode federation. And the maintained files info will be checkpointed
to HDFS ensure it's reliability. When SSM is restarted on other nodes, the data can be reloaded
from HDFS other than from NameNodes again. 

Any comment? Thanks!

> HDFS smart storage management
> -----------------------------
>                 Key: HDFS-7343
>                 URL: https://issues.apache.org/jira/browse/HDFS-7343
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kai Zheng
>            Assignee: Wei Zhou
>         Attachments: HDFSSmartStorageManagement-General-20170315.pdf, HDFS-Smart-Storage-Management.pdf,
HDFSSmartStorageManagement-Phase1-20170315.pdf, HDFS-Smart-Storage-Management-update.pdf,
> As discussed in HDFS-7285, it would be better to have a comprehensive and flexible storage
policy engine considering file attributes, metadata, data temperature, storage type, EC codec,
available hardware capabilities, user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution to provide
smart storage management service in order for convenient, intelligent and effective utilizing
of erasure coding or replicas, HDFS cache facility, HSM offering, and all kinds of tools (balancer,
mover, disk balancer and so on) in a large cluster.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message