hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7343) HDFS smart storage management
Date Fri, 24 Mar 2017 22:43:42 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941287#comment-15941287

Lei (Eddy) Xu commented on HDFS-7343:

Hi, [~drankye] and [~zhouwei]

Thanks for the updated design docs. It is a very good write up.

I have a few questions. 

* IIUC, SSM needs to maintain stats of each file / block in NN, which adds O(n) overhead to
JVM heap, where n is {# of blocks or # of files}.  Additionally, when SSM pulling these stats
from NN, what kind of overhead we are expecting? If SCM pulls the full block map / file namespace,
would that be a performance hit to NN?
* I think that in the general design, we should work on define a good interface time series
store for metrics, instead of specifying RocksDB. RocksDB might be a good implementation for
* In Figure 1 of general design, it is not clear to me why both SSM and NN need to persist
metrics in two _separated_ rocksdb? If NN needs to persist metrics to rocksdb, does that mean
both ANN and SNN in a HA setup need to persist them?  What about the HA of SSM?
* Rules. How stale or extensible the syntax will be? would the syntax be easy for other applications
to generate / parse / validate and etc? What would be your opinion of  using json or YAML
for the syntax? Would it be possible that when adding a rule, SSM can verify the rule based
on HSM/EC/SPS policies and etc?
* in Phase 1 design, {{file.accessCount(interval)}}. How to update the accessCount ? how many
samples of accessCount to be maintained the accessCount during an interval? What if SSM failed
to pull for the the accessCount?
* Maybe we can define SSM rules as _soft_ rules, while make HSM/EC and etc rules as hard rules?


> HDFS smart storage management
> -----------------------------
>                 Key: HDFS-7343
>                 URL: https://issues.apache.org/jira/browse/HDFS-7343
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kai Zheng
>            Assignee: Wei Zhou
>         Attachments: HDFSSmartStorageManagement-General-20170315.pdf, HDFS-Smart-Storage-Management.pdf,
HDFSSmartStorageManagement-Phase1-20170315.pdf, HDFS-Smart-Storage-Management-update.pdf,
> As discussed in HDFS-7285, it would be better to have a comprehensive and flexible storage
policy engine considering file attributes, metadata, data temperature, storage type, EC codec,
available hardware capabilities, user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution to provide
smart storage management service in order for convenient, intelligent and effective utilizing
of erasure coding or replicas, HDFS cache facility, HSM offering, and all kinds of tools (balancer,
mover, disk balancer and so on) in a large cluster.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message