Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 9 May 2017 15:26:05 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12647860.1368659396000.165334.1494343565392@Atlassian.JIRA>
In-Reply-To: <JIRA.12647860.1368659396000@Atlassian.JIRA>
References: <JIRA.12647860.1368659396000@Atlassian.JIRA> <JIRA.12647860.1368659396851@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to
 blobstore FileSystems
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 09 May 2017 15:26:14 -0000


     [ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-9565:
-----------------------------------
    Attachment: HADOOP-9565-010.patch

Patch 010: resetting everything and starting again with nothing but an API to query a filesystem for features. This is a WiP again, up for feedback.

* {{boolean FileSystem.hasFeature(Path, String)}} lets you see if the bit of an FS under a path has a feature
* Feature names in {{FileSystemFeatures}} are pretty much a copy and paste of {{org.apache.hadoop.fs.contract.ContractOptions}}.
* The base class mechanism for setting up feature resolution, {{FeatureResolver}} is driven off XML configuration files; with a setup mechanism which allows for subclasses to dynamically redefine the features, such as when setting up a local FS.

All the work of defining features is already in the contract tests; I've just pulled their declarations into the production hadoop-common, hadoop-hdfs-client JARs, not the test JARs, and modified the two migrated FS's contracts to load in the new XML file as well as the old one.

One big difference is in resolution of a feature: when a feature is looked for it first looks for the schema-specific option {{fs.hdfs.contract.supports-atomic-rename}} before falling back to look for {{fs.contract.supports-atomic-rename}}, which is where a default can be defined. 

This allows for  per-FS values to be set in core-default.xml, 

{code}
# default: listings work
fs.contract.supports-consistent-listing  = false
# but not S3A
fs.s3a.contract.supports-consistent-listing  = false
{code}

And overridden in core-site.xml, as it'd be in a deployment against WDC's servers

{code}
fs.s3a.contract.supports-consistent-listing  = true
{code}


And it permits S3A per-bucket override for deployments where only some buckets are consistent, such as "stevel-s3guard"

{code}
fs.s3a.bucket.stevel-s3guard.contract.supports-consistent-listing  = true
{code}

Although the feature resolver is driven purely by the loaded hash of config options, because the lookup is via the FS instance, it has the option of being dynamic: per-path, on-demand evaluation rather than on construction probes for OS type, ....

Also, the underlying feature resolver has a Java 8 optional API, so it can return yes/no/don't know, the idea being "no" can be normative.


> Add a Blobstore interface to add to blobstore FileSystems
> ---------------------------------------------------------
>
>                 Key: HADOOP-9565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9565
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3, fs/swift
>    Affects Versions: 2.6.0
>            Reporter: Steve Loughran
>            Assignee: Thomas Demoor
>         Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch, HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch, HADOOP-9565-006.patch, HADOOP-9565-008.patch, HADOOP-9565-010.patch, HADOOP-9565-branch-2-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really blobstores, with different atomicity and consistency guarantees, by adding a {{Blobstore}} interface to add to them. 
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that all blobstores implement at server-side copy operation as a substitute for rename.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org