hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@effectivemachines.com>
Subject Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk
Date Fri, 22 Jul 2016 15:48:25 GMT

Does any of this work actually help processes that sit outside of YARN?

> On Jul 21, 2016, at 12:29 PM, Sean Busbey <busbey@cloudera.com> wrote:
> 
> thanks for bringing this up! big +1 on upgrading dependencies for 3.0.
> 
> I have an updated patch for HADOOP-11804 ready to post this week. I've
> been updating HBase's master branch to try to make use of it, but
> could use some other reviews.
> 
> On Thu, Jul 21, 2016 at 4:30 AM, Tsuyoshi Ozawa <ozawa@apache.org> wrote:
>> Hi developers,
>> 
>> I'd like to discuss how to make an advance towards dependency
>> management in Apache Hadoop trunk code since there has been lots work
>> about updating dependencies in parallel. Summarizing recent works and
>> activities as follows:
>> 
>> 0) Currently, we have merged minimum update dependencies for making
>> Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
>> 1) After that, some people suggest that we should update the other
>> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
>> 2) In parallel, Sangjin and Sean are working on classpath isolation:
>> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
>> 
>> Main problems we try to solve in the activities above is as follows:
>> 
>> * 1) tries to solve dependency hell between user-level jar and
>> system(Hadoop)-level jar.
>> * 2) tries to solve updating old libraries.
>> 
>> IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries
>> to separate class loader between client-side dependencies and
>> server-side dependencies in Hadoop, so we can the change policy of
>> updating libraries after doing 2). We can also decide which libraries
>> can be shaded after 2).
>> 
>> Hence, IMHO, a straight way we should go to is doing 2 at first.
>> After that, we can update both client-side and server-side
>> dependencies based on new policy(maybe we should discuss what kind of
>> incompatibility is acceptable, and the others are not).
>> 
>> Thoughts?
>> 
>> Thanks,
>> - Tsuyoshi
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> 
> 
> 
> 
> -- 
> busbey
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


Mime
View raw message