spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dongjoon-hyun <...@git.apache.org>
Subject [GitHub] spark pull request #23119: [SPARK-25954][SS][FOLLOWUP][TEST-MAVEN] Add Zooke...
Date Thu, 22 Nov 2018 19:30:06 GMT
GitHub user dongjoon-hyun opened a pull request:

    https://github.com/apache/spark/pull/23119

    [SPARK-25954][SS][FOLLOWUP][TEST-MAVEN] Add Zookeeper 3.4.7 test dependency to Kafka modules

    ## What changes were proposed in this pull request?
    
    This is a followup of #23099 . After upgrading to Kafka 2.1.0, maven test fails due to
Zookeeper test dependency while sbt test succeeds.
    
    - [sbt test on master branch](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/5203/)
    - [maven test on master branch](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/5653/)
    
    The root cause is that the embedded Kafka server is using [Zookeepr 3.4.7 API](https://zookeeper.apache.org/doc/r3.4.7/api/org/apache/zookeeper/AsyncCallback.MultiCallback.html
    ) while Apache Spark provides Zookeeper 3.4.6. This PR adds a test dependency.
    
    ```
    KafkaMicroBatchV2SourceSuite:
    *** RUN ABORTED ***
    ...
    org.apache.spark.sql.kafka010.KafkaTestUtils.setupEmbeddedKafkaServer(KafkaTestUtils.scala:123)
      ...
      Cause: java.lang.ClassNotFoundException: org.apache.zookeeper.AsyncCallback$MultiCallback
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)
      at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348)
      at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
      at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
      at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2(KafkaTestUtils.scala:120)
      at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2$adapted(KafkaTestUtils.scala:116)
      ...
    ```
    
    ## How was this patch tested?
    
    Pass the maven Jenkins test.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dongjoon-hyun/spark SPARK-25954-2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/23119.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #23119
    
----
commit 9c5d7961406dccfb16685a18467e02f3ea8c1ce8
Author: Dongjoon Hyun <dongjoon@...>
Date:   2018-11-22T18:41:14Z

    [SPARK-25954][SS][FOLLOWUP] Add Zookeeper 3.4.7 test dependency to kafka-0-10 module
    
    https://zookeeper.apache.org/doc/r3.4.7/api/org/apache/zookeeper/AsyncCallback.MultiCallback.html
    
    KafkaMicroBatchV2SourceSuite:
    *** RUN ABORTED ***
      java.lang.NoClassDefFoundError: org/apache/zookeeper/AsyncCallback$MultiCallback
      at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)
      at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348)
      at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
      at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
      at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2(KafkaTestUtils.scala:120)
      at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2$adapted(KafkaTestUtils.scala:116)
      at org.apache.spark.util.Utils$.$anonfun$startServiceOnPort$2(Utils.scala:2221)
      at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:155)
      at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2213)
      at org.apache.spark.sql.kafka010.KafkaTestUtils.setupEmbeddedKafkaServer(KafkaTestUtils.scala:123)
      ...
      Cause: java.lang.ClassNotFoundException: org.apache.zookeeper.AsyncCallback$MultiCallback
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)
      at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348)
      at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
      at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
      at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2(KafkaTestUtils.scala:120)
      at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2$adapted(KafkaTestUtils.scala:116)
      ...
    
    a

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message