hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From larry mccay <lmc...@apache.org>
Subject Re: [DISCUSS] Branch Proposal: HADOOP 15407: ABFS
Date Tue, 15 May 2018 15:44:52 GMT
This seems like a reasonable and effective use of a feature branch and
branch committers to me.


On Tue, May 15, 2018 at 11:34 AM, Steve Loughran <stevel@hortonworks.com>
wrote:

> Hi
>
> Chris Douglas I and I've have a proposal for a short-lived feature branch
> for the Azure ABFS connector to go into the hadoop-azure package. This will
> connect to the new azure storage service, which will ultimately replace the
> one used by wasb. It's a big patch and, like all storage connectors, will
> inevitably take time to stabilize (i.e: nobody ever get seek() right, even
> when we think we have).
>
> Thomas & Esfandiar will do the coding: they've already done the paperwork.
> Chris, myself & anyone else interested can be involved in the review and
> testing.
>
> Comments?
>
> -------------
>
> The initial HADOOP-15407 patch contains a new filesystem client for the
> forthcoming Azure ABFS, which is intended to replace Azure WASB as the
> Azure storage layer. The patch is large, as it contains the replacement
> client, tests, and generated code.
>
> We propose a feature branch, so the module can be broken into salient,
> reviewable chunks. Internal constraints prevented this feature from being
> developed in Apache, so we want to ensure that all the code is discussed,
> maintainable, and documented by the community before it merges.
>
> To effect this, we also propose adding two developers as branch
> committers: Thomas Marquardt tmarq@microsoft.com<mailto:tma
> rq@microsoft.com> Esfandiar Manii esmanii@microsoft.com<mailto:e
> smanii@microsoft.com>
>
> Beyond normal feature branch activity and merge criteria for FS modules,
> we want to add another merge criterion for ABFS. Some of the client APIs
> are not GA. It seems reasonable to require that this client works with
> public endpoints before it merges to trunk.
>
> To test the Blob FS driver, Blob FS team (including Esfandiar Manii and
> Thomas Marquardt) in Azure Storage will need the MSDN subscription ID(s)
> for all reviewers who want to run the tests. The ABFS team will then
> whitelist the subscription ID(s) for the Blob FS Preview. At that time,
> future storage accounts created will have the Blob FS endpoint,
> <accountName>.dfs.core.windows.net<http://dfs.core.windows.net>, which
> the Blob FS driver relies on.
>
> This is a temporary state during the (current) Private Preview and the
> early phases of Public Preview. In a few months, the whitelisting will not
> be required and anyone will be able to create a storage account with access
> to the Blob FS endpoint.
>
> Thomas and Esfandiar have been active in the Hadoop project working on the
> WASB connector (see https://issues.apache.org/jira/browse/HADOOP-14552).
> They understand the processes and requirements of the software. Working on
> the branch directly will let them bring this significant feature into the
> hadoop-azure module without disrupting existing users.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message