Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
From: Mingliang Liu <liuml07@gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: [VOTE] Merge HADOOP-13345 (S3Guard feature branch)
Date: Sat, 19 Aug 2017 15:28:53 -0700
References: <CADZgEPKEEzsCCnfwhfd1zTMR=Y9-e0bi2eTHucJ7UJYkHk54zw@mail.gmail.com>
To: Hadoop Common <common-dev@hadoop.apache.org>
In-Reply-To: <CADZgEPKEEzsCCnfwhfd1zTMR=Y9-e0bi2eTHucJ7UJYkHk54zw@mail.gmail.com>
Message-Id: <A0DF483B-433A-46FF-AE06-677DBF3E01AE@gmail.com>
archived-at: Sat, 19 Aug 2017 22:29:08 -0000

+1 (non-binding)

I also worked on this project from start to finish and I really enjoyed =
the collaboration in community. The feature is to solve the very =
important and challenging consistency problem as stated in the design =
doc. All patches were reviewed by feature/trunk committers and we have =
been testing it with real world applications. Overall I think it is now =
production ready. Most contributors of this project are active in =
community and I believe the future work and code maintenance will be =
well addressed.

Thanks,

> On Aug 17, 2017, at 3:07 PM, Aaron Fabbri <fabbri@cloudera.com> wrote:
>=20
> Hello,
>=20
> I'd like to open a vote (7 days, ending August 24 at 3:10 PST) to =
merge the
> HADOOP-13345 feature branch into trunk.
>=20
> This branch contains the new S3Guard feature which adds metadata
> consistency features to the S3A client.  Formatted site documentation =
can
> be found here:
>=20
> =
https://github.com/apache/hadoop/blob/HADOOP-13345/hadoop-tools/hadoop-aws=
/src/site/markdown/tools/hadoop-aws/s3guard.md
>=20
> The current patch against trunk is posted here:
>=20
> https://issues.apache.org/jira/browse/HADOOP-13998
>=20
> The branch modifies the s3a portion of the hadoop-tools/hadoop-aws =
module:
>=20
> - The feature is off by default, and care has been taken to insure it =
has
> no impact when disabled.
> - S3Guard can be enabled with the production database which is backed =
by
> DynamoDB, or with a local, in-memory implementation that facilitates
> integration testing without having to pay for a database.
> - getFileStatus() as well as directory listing consistency has been
> implemented and thoroughly tested, including delete tracking.
> - Convenient Maven profiles for testing with and without S3Guard.
> - New failure injection code and integration tests that exercise it.  =
We
> use timers and a wrapper around the Amazon SDK client object to force
> consistency delays to occur.  This allows us to assert that S3Guard =
works
> as advertised.  This will be extended with more types of failure =
injection
> to continue hardening the S3A client.
>=20
> Outside of hadoop-tools/hadoop-aws's s3a directory there are some =
minor
> changes:
>=20
> - core-default.xml defaults and documentation for s3guard parameters.
> - A couple additional FS contract test cases around rename.
> - More goodies in LambdaTestUtils
> - A new CLI tool for inspecting and manipulating S3Guard features,
> including the backing MetadataStore database.
>=20
> This branch has seen extensive testing as well as use in production.  =
This
> branch makes significant improvements to S3A's test toolkit as well.
>=20
> Performance is typically on par with, and in some cases better than, =
the
> existing S3A code without S3Guard enabled.
>=20
> This feature was developed with contributions and feedback from many
> people.  I'd like to thank everyone who worked on HADOOP-13345 as well =
as
> all of those who contributed feedback and work on the original design
> document.
>=20
> This is the first major Apache Hadoop project I've worked on from =
start to
> finish, and I've really enjoyed it.  Please shout if I've missed =
anything
> important here or in the VOTE process.
>=20
> Cheers,
> Aaron Fabbri


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org