hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: Google Cloud Storage connector into Hadoop
Date Mon, 07 Dec 2015 23:54:08 GMT
Hi James,

This sounds great!  Thank you for considering contributing the code.

Just seconding what Haohui said, there is existing precedent for
alternative implementations of the Hadoop FileSystem in our codebase.  We
currently have similar plugins for S3 [1], Azure [2] and OpenStack Swift
[3].  Additionally, we have a suite of FileSystem contract tests [4].
These tests are designed to help developers of alternative file systems
assess how closely they match the semantics expected by Hadoop ecosystem

Many Hadoop users are accustomed to using HDFS instead of these
alternative file systems, so none of the alternatives are on the default
Hadoop classpath immediately after deployment.  Instead, the code for each
one is in a separate module under the "hadoop-tools" directory in the
source tree.  Users who need to use the alternative file systems take
extra steps post-deployment to add them to the classpath where necessary.
This achieves the dependency isolation needed.  For example, users who
never use the Azure plugin won't accidentally pick up a transitive
dependency on the Azure SDK jar.

I recommend taking a quick glance through the existing modules for S3,
Azure and OpenStack.  We'll likely ask that a new FileSystem
implementation follow the same patterns if feasible for consistency.  This
would include things like using the contract tests, having a provision to
execute tests both offline/mocked and live/integrated with the real
service and providing a documentation page that explains configuration for
end users.

For now, please feel free to file a HADOOP JIRA with your proposal.  We
can work out the details of all of this in discussion on that JIRA.

Something else to follow up on will be licensing concerns.  I see the
project already uses the Apache license, but it appears to be an existing
body of code initially developed at Google.  That might require a Software
Grant Agreement [5].  Again, this is something that can be hashed out in
discussion on the JIRA after you create it.

[2] http://hadoop.apache.org/docs/r2.7.1/hadoop-azure/index.html
[3] http://hadoop.apache.org/docs/r2.7.1/hadoop-openstack/index.html
[5] http://www.apache.org/licenses/

--Chris Nauroth

On 12/7/15, 3:10 PM, "Haohui Mai" <ricetons@gmail.com> wrote:

>Thanks for reaching out. It would be great to see this in the Hadoop
>In Hadoop we have AWS S3 support. IMO they address similar use cases
>thus I think that it should be relatively straightforward to adopt the
>The only catch in my head right now is to properly isolate dependency.
>Not only the code needs to be put into a separate module, but many
>Hadoop applications also depend on different versions of Guava. I
>think it might be a problem that needs some attentions at the very
>Please feel free to reach out if you have any other questions.
>On Mon, Dec 7, 2015 at 2:35 PM, James Malone
><jamesmalone@google.com.invalid> wrote:
>> Hello,
>> We're from a team within Google Cloud Platform focused on OSS and data
>> technologies, especially Hadoop (and Spark.) Before we cut a JIRA for
>> something we¹d like to do, we wanted to reach out to this list to ask a
>> quick questions, describe our proposed action, and check for any major
>> objections.
>> Proposed action:
>> We have a Hadoop connector[1] (more info[2]) for Google Cloud Storage
>> which we have been building and maintaining for some time. After we
>> up our code and tests to conform (to these[3] and other requirements) we
>> would like to contribute it to Hadoop. We have many customers using the
>> connector in high-throughput production Hadoop clusters; we¹d like to
>> it easier and faster to use Hadoop and GCS.
>> Timeline:
>> Presently, we are working on the beta of Google Cloud Dataproc[4] which
>> limits our time a bit, so we¹re targeting late Q1 2016 for creating a
>> issue and adapting our connector code as needed.
>> Our (quick) questions:
>> * Do we need to take any (non-coding) action for this beyond submitting
>> JIRA when we are ready?
>> * Are there any up-front concerns or questions which we can (or will
>> to) address?
>> Thank you!
>> James Malone
>> On behalf of the Google Big Data OSS Engineering Team
>> Links:
>> [1] - 
>> [2] - https://cloud.google.com/hadoop/google-cloud-storage-connector
>> [3] - 
>> [4] - https://cloud.google.com/dataproc

View raw message