hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8888) Support volumes in HDFS
Date Wed, 19 Aug 2015 23:41:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703974#comment-14703974

Andrew Wang commented on HDFS-8888:

I see encryption zones as the closest thing semantically to volumes right now because of the
rename restriction, and it's been incompatible with some applications like Hive (which we
fixed) and HDFS trash (which we haven't). Right now that pain is restricted to the subset
of HDFS users who are also using encryption, but volumes as a first-class citizen will bring
this into the spotlight. Volumes might be compelling enough to revisit the various rename
assumptions in our app stack, but need to think hard about the app changes that are required.

The motivations you've listed for the first phase of development reference simpler implementation
and management. Regarding implementation, we've already implemented the additional complexity
of doing it at the directory-level,  so what's the advantage of changing it up now? Management-wise,
I don't quite understand why it's easier to manage volumes vs. folders. You can treat some
folders as you would volumes and get the same properties, right?

The scalability motivations are more compelling to me since it's something we can't do now,
but there's still more vertical scalability work we can do first that preserves existing semantics.
Also if we want to pursue volumes vs. a true distributed namespace implementation which might
preserve existing semantics.

Finally, is this going to be linked with viewfs improvements? If volumes are a first-class
citizen and being added and removed all the time, it'd be nice to have a centralized mount
table rather than having to push out new client configs each time. Also need it to be able
to say, list the set of volumes, or automatically choosing a NN when provisioning or rebalancing

> Support volumes in HDFS
> -----------------------
>                 Key: HDFS-8888
>                 URL: https://issues.apache.org/jira/browse/HDFS-8888
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Haohui Mai
> There are multiple types of zones (e.g., snapshottable directories, encryption zones,
directories with quotas) which are conceptually close to namespace volumes in traditional
file systems.
> This jira proposes to introduce the concept of volume to simplify the implementation
of snapshots and encryption zones.

This message was sent by Atlassian JIRA

View raw message