hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
Date Tue, 09 May 2017 15:26:05 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated HADOOP-9565:
    Attachment: HADOOP-9565-010.patch

Patch 010: resetting everything and starting again with nothing but an API to query a filesystem
for features. This is a WiP again, up for feedback.

* {{boolean FileSystem.hasFeature(Path, String)}} lets you see if the bit of an FS under a
path has a feature
* Feature names in {{FileSystemFeatures}} are pretty much a copy and paste of {{org.apache.hadoop.fs.contract.ContractOptions}}.
* The base class mechanism for setting up feature resolution, {{FeatureResolver}} is driven
off XML configuration files; with a setup mechanism which allows for subclasses to dynamically
redefine the features, such as when setting up a local FS.

All the work of defining features is already in the contract tests; I've just pulled their
declarations into the production hadoop-common, hadoop-hdfs-client JARs, not the test JARs,
and modified the two migrated FS's contracts to load in the new XML file as well as the old

One big difference is in resolution of a feature: when a feature is looked for it first looks
for the schema-specific option {{fs.hdfs.contract.supports-atomic-rename}} before falling
back to look for {{fs.contract.supports-atomic-rename}}, which is where a default can be defined.

This allows for  per-FS values to be set in core-default.xml, 

# default: listings work
fs.contract.supports-consistent-listing  = false
# but not S3A
fs.s3a.contract.supports-consistent-listing  = false

And overridden in core-site.xml, as it'd be in a deployment against WDC's servers

fs.s3a.contract.supports-consistent-listing  = true

And it permits S3A per-bucket override for deployments where only some buckets are consistent,
such as "stevel-s3guard"

fs.s3a.bucket.stevel-s3guard.contract.supports-consistent-listing  = true

Although the feature resolver is driven purely by the loaded hash of config options, because
the lookup is via the FS instance, it has the option of being dynamic: per-path, on-demand
evaluation rather than on construction probes for OS type, ....

Also, the underlying feature resolver has a Java 8 optional API, so it can return yes/no/don't
know, the idea being "no" can be normative.

> Add a Blobstore interface to add to blobstore FileSystems
> ---------------------------------------------------------
>                 Key: HADOOP-9565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9565
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3, fs/swift
>    Affects Versions: 2.6.0
>            Reporter: Steve Loughran
>            Assignee: Thomas Demoor
>         Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch, HADOOP-9565-003.patch,
HADOOP-9565-004.patch, HADOOP-9565-005.patch, HADOOP-9565-006.patch, HADOOP-9565-008.patch,
HADOOP-9565-010.patch, HADOOP-9565-branch-2-007.patch
> We can make the fact that some {{FileSystem}} implementations are really blobstores,
with different atomicity and consistency guarantees, by adding a {{Blobstore}} interface to
add to them. 
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that all blobstores
implement at server-side copy operation as a substitute for rename.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message