hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
Date Wed, 29 Aug 2018 14:07:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596377#comment-16596377

Steve Loughran commented on HADOOP-15691:

BTW, I've realised that I could expose this not just through a new API, but simply as a set
of config options which you can query through FileSystem.getConf().getBoolean("fs.capabilities.fs.atomic-rename").
FS instances would add their list of capabilities in init(), so existing code can see if a
feature is there without needing to call the API.

This'd make it easier for apps like HBase, Tez & Spark to be able to probe for specific
features, e.g. whether they could skip the use of a temp file + rename. It would complicate
subclassed filesystems which changed the capabilites though, especially those which removed
them, and I'd also need to add some property declaring that the capabilities had been registered,
e.g. "fs.capabilities.registered"

> Add PathCapabilities to FS and FC to complement StreamCapabilities
> ------------------------------------------------------------------
>                 Key: HADOOP-15691
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15691
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to declare the
capabilities under the path of an FS
> This is needed for 
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside (slow-rename)
> * Anything else where the implementation semantics of an FS is so different caller apps
would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't hold, especially
for object stores —and to efficiently use them, callers need to be able to ask for specific

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message