hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sangjin Lee <sj...@apache.org>
Subject Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk
Date Mon, 25 Jul 2016 20:16:45 GMT
On Fri, Jul 22, 2016 at 5:15 PM, Allen Wittenauer <aw@effectivemachines.com>
wrote:

>
> But if I don't use ApplicationClassLoader, my java app is basically
> screwed then, right?
>

If we start upgrading the libraries aggressively, then it would also mean
that the ApplicationClassLoader should be more of the default than the
other way around (i.e. opt-out rather than opt-in). If we're not willing to
go there, then we cannot be too aggressive in upgrading libraries.

I'm not sure what you mean by "my java app is basically screwed", but if
you meant whether your java app would be OK if hadoop upgraded libraries
aggressively and you don't use the ApplicationClassLoader, then yes.


>
> Also:  right now, the non-Linux and/or non-x86 platforms have to supply
> their own leveldbjni jar (or at least the C level library?) in order to
> make YARN even functional.  How is that going to work with the class path
> manipulation?
>

First, the native libraries are orthogonal to this. They're not governed by
the java classpath.

For those platforms where users/admins need to provide their own LevelDB
libraries, the only requirement would be to add them to the
share/hadoop/.../lib directory. I don't think we would ask end users of the
clusters to bring in their own LevelDB library as it would not be an
end-user concern. I assume the administrators of clusters (still users but
not end users) would add it to the clusters. The classpath isolation
doesn't really have an impact on that.


>
> > On Jul 22, 2016, at 9:57 AM, Sangjin Lee <sjlee@apache.org> wrote:
> >
> > The work on HADOOP-13070 and the ApplicationClassLoader are generic and
> go beyond YARN. It can be used in any JVM that uses hadoop. The current use
> cases are MR containers, hadoop's RunJar (as in "hadoop jar"), and the YARN
> node manager auxiliary services. I'm not sure if that's what you were
> asking, but I hope it helps.
> >
> > Regards,
> > Sangjin
> >
> > On Fri, Jul 22, 2016 at 9:16 AM, Sean Busbey <busbey@cloudera.com>
> wrote:
> > My work on HADOOP-11804 *only* helps processes that sit outside of YARN.
> :)
> >
> > On Fri, Jul 22, 2016 at 10:48 AM, Allen Wittenauer
> > <aw@effectivemachines.com> wrote:
> > >
> > > Does any of this work actually help processes that sit outside of YARN?
> > >
> > >> On Jul 21, 2016, at 12:29 PM, Sean Busbey <busbey@cloudera.com>
> wrote:
> > >>
> > >> thanks for bringing this up! big +1 on upgrading dependencies for 3.0.
> > >>
> > >> I have an updated patch for HADOOP-11804 ready to post this week. I've
> > >> been updating HBase's master branch to try to make use of it, but
> > >> could use some other reviews.
> > >>
> > >> On Thu, Jul 21, 2016 at 4:30 AM, Tsuyoshi Ozawa <ozawa@apache.org>
> wrote:
> > >>> Hi developers,
> > >>>
> > >>> I'd like to discuss how to make an advance towards dependency
> > >>> management in Apache Hadoop trunk code since there has been lots work
> > >>> about updating dependencies in parallel. Summarizing recent works and
> > >>> activities as follows:
> > >>>
> > >>> 0) Currently, we have merged minimum update dependencies for making
> > >>> Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
> > >>> 1) After that, some people suggest that we should update the other
> > >>> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
> > >>> 2) In parallel, Sangjin and Sean are working on classpath isolation:
> > >>> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
> > >>>
> > >>> Main problems we try to solve in the activities above is as follows:
> > >>>
> > >>> * 1) tries to solve dependency hell between user-level jar and
> > >>> system(Hadoop)-level jar.
> > >>> * 2) tries to solve updating old libraries.
> > >>>
> > >>> IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries
> > >>> to separate class loader between client-side dependencies and
> > >>> server-side dependencies in Hadoop, so we can the change policy of
> > >>> updating libraries after doing 2). We can also decide which libraries
> > >>> can be shaded after 2).
> > >>>
> > >>> Hence, IMHO, a straight way we should go to is doing 2 at first.
> > >>> After that, we can update both client-side and server-side
> > >>> dependencies based on new policy(maybe we should discuss what kind
of
> > >>> incompatibility is acceptable, and the others are not).
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> Thanks,
> > >>> - Tsuyoshi
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > >>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> busbey
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> >
> >
> >
> > --
> > busbey
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message