hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13952) tools dependency hooks are throwing errors
Date Fri, 04 Aug 2017 17:31:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114676#comment-16114676
] 

Allen Wittenauer commented on HADOOP-13952:
-------------------------------------------

bq. anyone not using HDFS for storage shouldn't be pulling in other HDFS-specific things into
their classpath. 

>From a logistical stand point, this is correct.  However, as you observe:

bq. Although I'd expect they actually do that in practice now?

In practice?  

* 0.x-2.x always forces subproject directories into the classpath. Don't want them? Too bad.
* 3.x at least looks to see if these directories exist and allows function redefinition to
cut them out.
* I don't know of a single distribution that *doesn't* include HDFS or YARN (yes, including
MapR).  In fact, I'm not even sure it's possible to run Hadoop without HDFS given things like
hdfs getconf have no equivalent in common.

It's pretty much a guarantee these jars are already in the classpath. It's worth pointing
out that if they weren't, these tools likely wouldn't be working now. :)

IMO, if someone is going to go through the effort of ripping out HDFS, then changing the hadoop-tools
dependency profile is the least of their concerns.  That said... we should probably at least
add comments to the profile what the implied dependencies are though to make that easier.


> tools dependency hooks are throwing errors
> ------------------------------------------
>
>                 Key: HADOOP-13952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13952
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Allen Wittenauer
>            Assignee: Sean Mackrory
>            Priority: Critical
>         Attachments: HADOOP-13952.preview.patch
>
>
> During build, we are throwing these errors:
> {code}
> ERROR: hadoop-aliyun has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> ERROR: hadoop-archive-logs has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-archives has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aws has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-azure has missing dependencies: jetty-util-ajax-9.3.11.v20160721.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
> ERROR: hadoop-extras has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-gridmix has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-kafka has missing dependencies: lz4-1.2.0.jar
> ERROR: hadoop-kafka has missing dependencies: kafka-clients-0.8.2.1.jar
> ERROR: hadoop-openstack has missing dependencies: commons-httpclient-3.1.jar
> ERROR: hadoop-rumen has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: metrics-core-3.0.1.jar
> ERROR: hadoop-streaming has missing dependencies: jasper-compiler-5.5.23.jar
> {code}
> Likely a variety of reasons for the failures.  Kafka is HADOOP-12556, but others need
to be investigated.  Probably just need to look at more than just common/lib in dist-tools-hooks-maker
now that shading has gone in.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message