hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9534) Add CLI command to clear storage policy from a path.
Date Wed, 20 Jan 2016 21:01:40 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15109393#comment-15109393

Chris Nauroth commented on HDFS-9534:

Here are more details on the user scenario that prompted me to file this issue.  Someone set
a storage policy on a directory by mistake.  They were looking for a way to remove it, but
we don't currently have an easy way to do that.  Current workarounds are either to create
a whole new directory and move everything under that new directory, or explicitly set it back
to the parent directory's effective storage policy.

Note that the latter workaround is not quite the same semantics as removing the storage policy.
 In that case, storage policy changes higher in the tree would no longer propagate automatically
down to the child, because the child would still have an explicit policy of its own acting
as an override.  I think the effect of this new command should be to revert the storage policy
on the inode, not explicitly override it to the default, so that it restores the previous
behavior of that inode inheriting its policy from higher in the tree.  That would satisfy
the "undo" requirement for this user.

bq. Thus I do not think we need an explicit policy named "UNSPECIFIED".

I agree.  The patch as currently written would expose UNSPECIFIED in the API footprint, so
that callers could write code like {{fileSystem.setStoragePolicy(new Path("/foo"), "UNSPECIFIED");}}.
 I don't think that's desired.  Instead, I've always thought of UNSPECIFIED as an internal
implementation detail.  It appears that the only reason the name was added to {{BlockStoragePolicySuite}}
is so that it would be easier to reuse existing methods in {{FSDirAttrOp}}.  (Let me know
if I'm missing something.)  I'd prefer to avoid exposing UNSPECIFIED to clients, even if it
means we need a bit of extra code in {{FSDirAttrOp}} to support setting the policy without
access to a name defined in {{BlockStoragePolicySuite}}.

bq. If we have set a storage policy on "/foo", then should we allow the user to apply the
command on "/foo/bar" if bar is not associated with any explicit policy?

I think it's OK to let this happen and return success from the RPC, but it's effectively a
no-op.  That makes idempotence easy.

bq. Or what if we have nested policy settings?

I had been thinking that the remove operation applies just to the specific named inode, and
it would not directly alter anything lower in the tree.  (Of course, it's possible that removing
the storage policy from an inode causes its children to start inheriting different storage
policy settings from higher in the tree.)

> Add CLI command to clear storage policy from a path.
> ----------------------------------------------------
>                 Key: HDFS-9534
>                 URL: https://issues.apache.org/jira/browse/HDFS-9534
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: tools
>            Reporter: Chris Nauroth
>            Assignee: Xiaobing Zhou
>         Attachments: HDFS-9534.001.patch
> The {{hdfs storagepolicies}} command has sub-commands for {{-setStoragePolicy}} and {{-getStoragePolicy}}
on a path.  However, there is no {{-removeStoragePolicy}} to remove a previously set storage
policy on a path.

This message was sent by Atlassian JIRA

View raw message