hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk
Date Fri, 22 Jul 2016 16:16:13 GMT
My work on HADOOP-11804 *only* helps processes that sit outside of YARN. :)

On Fri, Jul 22, 2016 at 10:48 AM, Allen Wittenauer
<aw@effectivemachines.com> wrote:
>
> Does any of this work actually help processes that sit outside of YARN?
>
>> On Jul 21, 2016, at 12:29 PM, Sean Busbey <busbey@cloudera.com> wrote:
>>
>> thanks for bringing this up! big +1 on upgrading dependencies for 3.0.
>>
>> I have an updated patch for HADOOP-11804 ready to post this week. I've
>> been updating HBase's master branch to try to make use of it, but
>> could use some other reviews.
>>
>> On Thu, Jul 21, 2016 at 4:30 AM, Tsuyoshi Ozawa <ozawa@apache.org> wrote:
>>> Hi developers,
>>>
>>> I'd like to discuss how to make an advance towards dependency
>>> management in Apache Hadoop trunk code since there has been lots work
>>> about updating dependencies in parallel. Summarizing recent works and
>>> activities as follows:
>>>
>>> 0) Currently, we have merged minimum update dependencies for making
>>> Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
>>> 1) After that, some people suggest that we should update the other
>>> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
>>> 2) In parallel, Sangjin and Sean are working on classpath isolation:
>>> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
>>>
>>> Main problems we try to solve in the activities above is as follows:
>>>
>>> * 1) tries to solve dependency hell between user-level jar and
>>> system(Hadoop)-level jar.
>>> * 2) tries to solve updating old libraries.
>>>
>>> IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries
>>> to separate class loader between client-side dependencies and
>>> server-side dependencies in Hadoop, so we can the change policy of
>>> updating libraries after doing 2). We can also decide which libraries
>>> can be shaded after 2).
>>>
>>> Hence, IMHO, a straight way we should go to is doing 2 at first.
>>> After that, we can update both client-side and server-side
>>> dependencies based on new policy(maybe we should discuss what kind of
>>> incompatibility is acceptable, and the others are not).
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> - Tsuyoshi
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>
>>
>>
>> --
>> busbey
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>



-- 
busbey

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


Mime
View raw message