hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mate Szalay-Beko (Jira)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-16765) Fix curator dependencies for gradle projects using hadoop-minicluster
Date Mon, 16 Dec 2019 08:16:00 GMT
Mate Szalay-Beko created HADOOP-16765:
-----------------------------------------

             Summary: Fix curator dependencies for gradle projects using hadoop-minicluster
                 Key: HADOOP-16765
                 URL: https://issues.apache.org/jira/browse/HADOOP-16765
             Project: Hadoop Common
          Issue Type: Bug
          Components: common
            Reporter: Mate Szalay-Beko


*The Problem:*

The Kudu unit tests that use the `MiniDFSCluster` are broken due to a guava dependency issue
in the `hadoop-minicluster` module.
{code:java}
java.lang.NoSuchMethodError: com.google.common.util.concurrent.Futures.addCallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureCallback;)V
        at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.addResultCachingCallback(ThrottledAsyncChecker.java:167)
        at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:156)
        at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:166)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2794)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2709)
        at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1669)
        at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:911)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:518)
        at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:477)
        at org.apache.kudu.backup.HDFSTestKuduBackupLister.setUp(TestKuduBackupLister.scala:216)
{code}
The issue in that change is that even though Guava was excluded from the `curator-client`
module, just below that the `curator-framework` module is defined and doesn't exclude Gauva:
[https://github.com/apache/hadoop/blob/fccccc97034b29243a0509633849de55aa734859/hadoop-project/pom.xml#L1391-L1414]

This causes Guava 27.0.1-jre to be pulled in instead of Guava 11.0.2 defined by Hadoop:
{noformat}
+--- org.apache.hadoop:hadoop-minicluster:3.1.1.7.1.0.0-SNAPSHOT
|    +--- org.apache.hadoop:hadoop-common:3.1.1.7.1.0.0-SNAPSHOT
|    |    +--- org.apache.hadoop:hadoop-annotations:3.1.1.7.1.0.0-SNAPSHOT
|    |    +--- com.google.guava:guava:11.0.2 -> 27.0.1-jre
{noformat}
{noformat}
+--- org.apache.curator:curator-framework:4.2.0
|    \--- org.apache.curator:curator-client:4.2.0
|         +--- org.apache.zookeeper:zookeeper:3.5.4-beta -> 3.5.5.7.1.0.0-SNAPSHOT (*)
|         +--- com.google.guava:guava:27.0.1-jre (*)
|         \--- org.slf4j:slf4j-api:1.7.25{noformat}
 

*The root cause:*

I was able to reproduce this issue with some dummy projects, see [https://github.com/symat/transitive-dependency-test]

It seems that gradle behaves in this case differently than maven. If someone is using maven,
then he will not see this problem, as the exclude rules defined for the {{curator-client}} will
be enforced even if the {{curator-client}} comes transitively through the {{curator-framework}}.
While using the hadoop-minicluster in a gradle project will lead to this problem (unless extra
excludes / dependencies gets defined in the gradle project).

*The proposed solution* is to add the exclude rules for all Curator dependencies, preventing
other gradle projects using Hadoop from breaking because of the Curator upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message