hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13952) tools dependency hooks are throwing errors
Date Fri, 04 Aug 2017 14:43:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114453#comment-16114453
] 

Sean Mackrory commented on HADOOP-13952:
----------------------------------------

Changing findfileindir in dist-layout-stitching to only omit copying files if they're under
share/hadoop/common (rather than anywhere under share) fixes the problems, but has some other
effects as well. It increases total tarball size by 4MB and adds the following files:

{code}
./hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/mapreduce/lib: # dir didn't exist at all before
-rw-rw-r--. 1 sean sean    4467 aopalliance-1.0.jar
-rw-rw-r--. 1 sean sean  668235 guice-4.0.jar
-rw-rw-r--. 1 sean sean   76983 guice-servlet-4.0.jar
-rw-rw-r--. 1 sean sean    2497 javax.inject-1.jar
-rw-rw-r--. 1 sean sean   16151 jersey-guice-1.19.jar
-rw-rw-r--. 1 sean sean 1045744 leveldbjni-all-1.8.jar

./hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/tools/lib:
-rw-rw-r--. 1 sean sean    38302 jetty-util-ajax-9.3.11.v20160721.jar
-rw-rw-r--. 1 sean sean    85449 metrics-core-3.0.1.jar
-rw-rw-r--. 1 sean sean   319099 okhttp-2.4.0.jar
-rw-rw-r--. 1 sean sean    64661 okio-1.4.0.jar

./hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/yarn/lib:
-rw-rw-r--. 1 sean sean   38302 jetty-util-ajax-9.3.11.v20160721.jar
-rw-rw-r--. 1 sean sean 1045744 leveldbjni-all-1.8.jar
-rw-rw-r--. 1 sean sean 1779991 netty-all-4.0.23.Final.jar
{code}

This seems like the correct solution to me, although it perhaps indicates a few other dependencies
could be moved to common since they are in fact shared between multiple things apparently.
The alternative is to not consider dependencies missing if they're provided by HDFS, but having
tools share classpath with HDFS doesn't feel right. Although I'd expect they actually do that
in practice now?

> tools dependency hooks are throwing errors
> ------------------------------------------
>
>                 Key: HADOOP-13952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13952
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Allen Wittenauer
>            Assignee: Sean Mackrory
>            Priority: Critical
>         Attachments: HADOOP-13952.preview.patch
>
>
> During build, we are throwing these errors:
> {code}
> ERROR: hadoop-aliyun has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> ERROR: hadoop-archive-logs has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-archives has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aws has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-azure has missing dependencies: jetty-util-ajax-9.3.11.v20160721.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
> ERROR: hadoop-extras has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-gridmix has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-kafka has missing dependencies: lz4-1.2.0.jar
> ERROR: hadoop-kafka has missing dependencies: kafka-clients-0.8.2.1.jar
> ERROR: hadoop-openstack has missing dependencies: commons-httpclient-3.1.jar
> ERROR: hadoop-rumen has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: metrics-core-3.0.1.jar
> ERROR: hadoop-streaming has missing dependencies: jasper-compiler-5.5.23.jar
> {code}
> Likely a variety of reasons for the failures.  Kafka is HADOOP-12556, but others need
to be investigated.  Probably just need to look at more than just common/lib in dist-tools-hooks-maker
now that shading has gone in.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message