hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7190) Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath
Date Thu, 28 Sep 2017 20:19:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184806#comment-16184806
] 

Varun Saxena commented on YARN-7190:
------------------------------------

I haven't cleaned up findbugs-annotations and another unnecessary jar above. The result above
was merely to show possible segregation scheme. These jars are brought in as a hbase dependency.

I think breaking it up into server and client jars should be done across modules, not just
YARN. I was finding it weird to create a folder like {{share/hadoop/yarn/server}} and place
only the YARN timelineservice specific jars there and not the RM,NM ones. And if we place
RM, NM jars there it would break compatibility. So went with scheme above.

One suggestion from Vrushali during our weekly call was that we can place timelineservice
jar in share/hadoop/yarn/timelineservice folder and timelineservice hbase implementation jars
in share/hadoop/yarn/timelineservice-hbase folders. However, latter may make only sense if
we can based on timeline writer/reader implementation exclude this from NM/RM/reader classpath.
Not sure if this would be feasible.

The first suggestion sounds good to me. So now, I plan to create a folder {{share/hadoop/yarn/timelineservice}}
and {{share/hadoop/yarn/timelineservice/lib}} to place timeline service specific jars and
their dependencies.
Old jars would remain as it is in their respective locations.

Thoughts?

> Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-7190
>                 URL: https://issues.apache.org/jira/browse/YARN-7190
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineclient, timelinereader, timelineserver
>            Reporter: Vrushali C
>            Assignee: Varun Saxena
>
> [~jlowe] had a good observation about the user classpath getting extra jars in hadoop
2.x brought in with TSv2.  If users start picking up Hadoop 2,x's version of HBase jars instead
of the ones they shipped with their job, it could be a problem.
> So when TSv2 is to be used in 2,x, the hbase related jars should come into only the NM
classpath not the user classpath.
> Here is a list of some jars
> {code}
> commons-csv-1.0.jar
> commons-el-1.0.jar
> commons-httpclient-3.1.jar
> disruptor-3.3.0.jar
> findbugs-annotations-1.3.9-1.jar
> hbase-annotations-1.2.6.jar
> hbase-client-1.2.6.jar
> hbase-common-1.2.6.jar
> hbase-hadoop2-compat-1.2.6.jar
> hbase-hadoop-compat-1.2.6.jar
> hbase-prefix-tree-1.2.6.jar
> hbase-procedure-1.2.6.jar
> hbase-protocol-1.2.6.jar
> hbase-server-1.2.6.jar
> htrace-core-3.1.0-incubating.jar
> jamon-runtime-2.4.1.jar
> jasper-compiler-5.5.23.jar
> jasper-runtime-5.5.23.jar
> jcodings-1.0.8.jar
> joni-2.1.2.jar
> jsp-2.1-6.1.14.jar
> jsp-api-2.1-6.1.14.jar
> jsr311-api-1.1.1.jar
> metrics-core-2.2.0.jar
> servlet-api-2.5-6.1.14.jar
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message