hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@alumni.cmu.edu>
Subject Re: Guava
Date Mon, 10 Nov 2014 18:11:41 GMT
I'm usually an advocate for getting rid of unnecessary dependencies
(cough, jetty, cough), but a lot of the things in Guava are really
useful.

Immutable collections, BiMap, Multisets, Arrays#asList, the stuff for
writing hashCode() and equals(), String#Joiner, the list goes on.  We
particularly use the Cache/CacheBuilder stuff a lot in HDFS to get
maps with LRU eviction without writing a lot of boilerplate.  The QJM
stuff uses ListenableFuture a lot, although perhaps we could come up
with our own equivalent for that.

On Mon, Nov 10, 2014 at 9:26 AM, Alejandro Abdelnur <tucu00@gmail.com> wrote:
> IMO we should:
>
> 1* have a clean and thin client API JAR (which does not drag any 3rd party
> dependencies, or a well defined small set -i.e. slf4j & log4j-)
> 2* have a client implementation that uses a classloader to isolate client
> impl 3rd party deps from app dependencies.
>
> #2 can be done using a stock URLClassLoader (i would just subclass it to
> forbid packages in the API JAR and exposed 3rd parties to be loaded from
> the app JAR)
>
> #1 is the tricky thing as our current API modules don't have a clean
> API/impl separation.
>
> thx
> PS: If folks are interested in pursing this, I can put together a prototype
> of how  #2 would work (I don't think it will be more than 200 lines of code)

Absolutely, I agree that we should not be using Guava types in public
APIs.  Guava has not been very responsible with backwards
compatibility, that much is clear.

A client / server jar separation is an interesting idea.  But then we
still have to get rid of Guava and other library deps in the client
jars.  I think it would be more work than it seems.  For example, the
HDFS client uses Guava Cache a lot, so we'd have to write our own
version of this.

Can't we just shade this stuff?  Has anyone tried shading Hadoop's Guava?

best,
Colin


>
>
> On Mon, Nov 10, 2014 at 5:18 AM, Steve Loughran <stevel@hortonworks.com>
> wrote:
>
>> Yes, Guava is a constant pain; there's lots of open JIRAs related to it, as
>> its the one we can't seamlessly upgrade. Not unless we do our own fork and
>> reinsert the missing classes.
>>
>> The most common uses in the code are
>>
>> @VisibleForTesting (easily replicated)
>> and the Precondition.check() operations
>>
>> The latter is also easily swapped out, and we could even add the check they
>> forgot:
>> Preconditions.checkArgNotNull(argname, arg)
>>
>>
>> These are easy; its the more complex data structures that matter more.
>>
>> I think for Hadoop 2.7 & java 7 we need to look at this problem and do
>> something. Even if we continue to ship Guava 11 so that the HBase team
>> don't send any (more) death threats, we can/should rework Hadoop to build
>> and run against Guava 16+ too. That's needed to fix some of the recent java
>> 7/8+ changes.
>>
>> -Everything in v11 dropped from v16 MUST  to be implemented with our own
>> versions.
>> -anything tagged as deprecated in 11+ SHOULD be replaced by newer stuff,
>> wherever possible.
>>
>> I think for 2.7+ we should add some new profiles to the POM, for Java 8 and
>> 9 alongside the new baseline java 7. For those later versions we could
>> perhaps mandate Guava 16.
>>
>>
>>
>> On 10 November 2014 00:42, Arun C Murthy <acm@hortonworks.com> wrote:
>>
>> > … has been a constant pain w.r.t compatibility etc.
>> >
>> > Should we consider adopting a policy to not use guava in
>> Common/HDFS/YARN?
>> >
>> > MR doesn't matter too much since it's application-side issue, it does
>> hurt
>> > end-users though since they still might want a newer guava-version, but
>> at
>> > least they can modify MR.
>> >
>> > Thoughts?
>> >
>> > thanks,
>> > Arun
>> >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>

Mime
View raw message