hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10473) Allow only suitable storage policies to be set on striped files
Date Wed, 08 Jun 2016 00:13:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319754#comment-15319754

Jing Zhao commented on HDFS-10473:

Thanks for working on this, Uma! Could you please explain more on why "existing storage policies
are not suitable for striped layout files" ? My understanding is policies like "WARM" and
"ONE_SSD" are mainly targeting replication (since they're mainly setting specific storage
type for the first replica) thus are not suitable. Could you please confirm it?

For the patch, storage policies are mainly set on directories (in fact to set storage policies
on files is not recommended), and we allow moving EC files across EC directory boundaries.
Therefore it is not possible to disallow setting storage policies on striped file in O(1)
time complexity. Looks like the changes on the NN side may be unnecessary here. We only need
to let Mover ignore striped files for now.

However, this change may cause other issue. Since currently the main use case for EC is cold
data, it is very natural for a customer to set a directory as EC, and set COLD storage policy
on the directory. In this way all the EC files created later under this directory will be
placed on Archival storages. We should keep this semantic since this is a very strong use
case, but in the meanwhile, disabling Mover for EC files will conflict with this semantic:
i.e., we recognize storage policies during file creation but not afterwards.

Therefore, currently I think we can either 1) make no changes at all and depend on admin to
make the correct decision while setting EC and storage policies, or 2) have a long term plan
to fix the issue completely. For #2 maybe the best way is to bring in Volume concept, since
if we have different settings on nested directories we will have to scan the subtree for validation.

> Allow only suitable storage policies to be set on striped files
> ---------------------------------------------------------------
>                 Key: HDFS-10473
>                 URL: https://issues.apache.org/jira/browse/HDFS-10473
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HDFS-10473-01.patch
> Currently existing storage policies are not suitable for striped layout files.
> This JIRA proposes to reject setting storage policy on striped files.
> Another thought is to allow only suitable storage polices like ALL_SSD.
> Since the major use case of EC is for cold data, this may not be at high importance.
So, I am ok to reject setting storage policy on striped files at this stage. Please suggest
if others have some thoughts on this.
> Thanks [~zhz] for offline discussion on this.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message