From user-return-20504-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Jun 11 13:26:39 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BCE5C180647 for ; Mon, 11 Jun 2018 13:26:38 +0200 (CEST) Received: (qmail 99534 invoked by uid 500); 11 Jun 2018 11:26:37 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 99512 invoked by uid 99); 11 Jun 2018 11:26:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jun 2018 11:26:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 0BD02C00DB for ; Mon, 11 Jun 2018 11:26:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.282 X-Spam-Level: *** X-Spam-Status: No, score=3.282 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=data-artisans-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id Ag5E4eAdx0dX for ; Mon, 11 Jun 2018 11:26:34 +0000 (UTC) Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id B2D645F520 for ; Mon, 11 Jun 2018 11:26:33 +0000 (UTC) Received: by mail-wm0-f65.google.com with SMTP id v16-v6so14038772wmh.5 for ; Mon, 11 Jun 2018 04:26:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=data-artisans-com.20150623.gappssmtp.com; s=20150623; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=n3LVomMh4LV/ZonVS573UUxtFmUqcjhwMH4EphAYAR4=; b=a8uYSJ+4eFc9In9f5XxVz2U4+VVBTBitevhpz01k3nnI357t8yl2kdE432QptOBAU7 bDWCL362HJnoWqIga0rVgOP1H0EMLMjGQaR/QKZQ8yPrqxtpuC5HLoGgrMRYTzz+Vf8L enyh6HKkOsLH2cMTA+9K1SCdJtReDfl6MVRTJCz+t7lkpXg9iEXPhzHYMoHZTjPuN1eC IMRKGF9HteG7zeLosT+HOfPa44DpFf6C0PUku8sVw51ESyxSawil4jtuWjq3MXR0eejx MdaqrqK9Qg241z4YXJ7ls1vzc3mxwzuUbbFgASJF+lb73z+SwXmvXLawZx0OqOBIPR0+ 0+JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=n3LVomMh4LV/ZonVS573UUxtFmUqcjhwMH4EphAYAR4=; b=sZm3wh4q88leK4D2QYXJYUg1hpqW5gkq63SVvxuh+WOImKaI2429o/wOcdRi9Buaac FZW91st3Yhs79M5RrcTMCrtlVMh8UsX7oaRt5+TkLpTCmLC1lTt0z6Quaufp24ILBCMR 6MOGgvlNbOCnepZVl2uQCfkg0Q+Fbekl5XNE4MiQv6ip/Plr/ofgD1WxCsn+F3z6MCnJ 0K6BWp6vSjGbDgFv88RcSwHx033s9lQL1xYEaM1gD6S1n66yH5M6JOgsrjnICYOAHmNp ElWH0i9rJcBXgsl4FQ43vjo53YOIc8k/AfJrdL+t72gD/xY+pciZl6K0dYomnImGKdpR 63Sg== X-Gm-Message-State: APt69E0Kdi6AHFsGr0DPoxcWfknGHxVvO8HBiXojuzuBn9EvurPzduVZ V2osXtZdT8hdg4PQxx6X7vlCCQ== X-Google-Smtp-Source: ADUXVKIqCEmsQ8zMyd9IP4IDpMQ85+J/YF+3ra7UtnjKmn+m4AMDDUKrHnBiRsPsBw3KoXFlNtgnuQ== X-Received: by 2002:a1c:30d7:: with SMTP id w206-v6mr7360586wmw.153.1528716393206; Mon, 11 Jun 2018 04:26:33 -0700 (PDT) Received: from piotrs-mbp.fritz.box (dslb-002-205-083-036.002.205.pools.vodafone-ip.de. [2.205.83.36]) by smtp.gmail.com with ESMTPSA id g11-v6sm10153195wrr.46.2018.06.11.04.26.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 Jun 2018 04:26:31 -0700 (PDT) From: Piotr Nowojski Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_5BA9E806-2EB4-474F-9CF2-8D1308D89760" Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\)) Subject: Re: Akka version conflict running on Flink cluster Date: Mon, 11 Jun 2018 13:26:30 +0200 In-Reply-To: Cc: "user@flink.apache.org" To: Wouter Zorgdrager References: X-Mailer: Apple Mail (2.3445.6.18) --Apple-Mail=_5BA9E806-2EB4-474F-9CF2-8D1308D89760 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, Please take a look on this thread first: = http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Akka-H= ttp-used-in-custom-RichSourceFunction-td20314.html = Piotrek > On 11 Jun 2018, at 11:16, Wouter Zorgdrager = wrote: >=20 > Hi, >=20 > I think I'm running into an Akka version conflict when running a Flink = job on a cluster. >=20 > The current situation: > - Flink cluster on Flink 1.4.2 (using Docker) > - Flink job which uses twitter4s [1] library and Akka version 2.5.8 >=20 > In my Flink job I try to 'shutdown' an Akka actor from the twitter4s = library. > This results in a whole taskmanager crashing with the following = stacktrace: >=20 > taskrunner_1 | 2018-06-11 09:03:14,454 INFO = org.apache.flink.runtime.taskmanager.TaskManager - = Un-registering task and sending final execution state CANCELED to = JobManager for task Source: Custom Source -> Sink: Unnamed = (0ba7f7f259eee06fe2f7d783c868179b) > taskrunner_1 | Uncaught error from thread = [twitter4s-streaming-akka.actor.default-dispatcher-288]: loader = constraint violation: when resolving method = "akka.actor.ActorCell$$anonfun$3.(Lakka/actor/ActorCell;)V" the = class loader (instance of = org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$= ChildFirstClassLoader) of the current class, akka/actor/ActorCell, and = the class loader (instance of sun/misc/Launcher$AppClassLoader) for the = method's defining class, akka/actor/ActorCell$$anonfun$3, have different = Class objects for the type akka/actor/ActorCell used in the signature, = shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for = for ActorSystem[twitter4s-streaming] > taskrunner_1 | java.lang.LinkageError: loader constraint violation: = when resolving method = "akka.actor.ActorCell$$anonfun$3.(Lakka/actor/ActorCell;)V" the = class loader (instance of = org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$= ChildFirstClassLoader) of the current class, akka/actor/ActorCell, and = the class loader (instance of sun/misc/Launcher$AppClassLoader) for the = method's defining class, akka/actor/ActorCell$$anonfun$3, have different = Class objects for the type akka/actor/ActorCell used in the signature > taskrunner_1 | at = akka.actor.ActorCell.invoke(ActorCell.scala:499) > taskrunner_1 | at = akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) > taskrunner_1 | at akka.dispatch.Mailbox.run(Mailbox.scala:224) > taskrunner_1 | at akka.dispatch.Mailbox.exec(Mailbox.scala:234) > taskrunner_1 | at = akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > taskrunner_1 | at = akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:13= 39) > taskrunner_1 | at = akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > taskrunner_1 | at = akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:= 107) > taskrunner_1 | 2018-06-11 09:03:14,984 INFO = org.apache.flink.runtime.blob.PermanentBlobCache - Shutting = down BLOB cache > taskrunner_1 | 2018-06-11 09:03:14,985 INFO = org.apache.flink.runtime.blob.TransientBlobCache - Shutting = down BLOB cache > taskrunner_1 | Exception in thread = "twitter4s-streaming-shutdown-hook-1" java.lang.NoClassDefFoundError: = akka/actor/CoordinatedShutdown$$anonfun$totalTimeout$1 > taskrunner_1 | at = akka.actor.CoordinatedShutdown.totalTimeout(CoordinatedShutdown.scala:515)= > taskrunner_1 | at = akka.actor.CoordinatedShutdown$$anonfun$initJvmHook$1.apply(CoordinatedShu= tdown.scala:217) > taskrunner_1 | at = akka.actor.CoordinatedShutdown$$anon$2.run(CoordinatedShutdown.scala:547) > taskrunner_1 | Caused by: java.lang.ClassNotFoundException: = akka.actor.CoordinatedShutdown$$anonfun$totalTimeout$1 > taskrunner_1 | at = java.net.URLClassLoader.findClass(URLClassLoader.java:381) > taskrunner_1 | at = java.lang.ClassLoader.loadClass(ClassLoader.java:424) > taskrunner_1 | at = org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$= ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128) > taskrunner_1 | at = java.lang.ClassLoader.loadClass(ClassLoader.java:357) > taskrunner_1 | ... 3 more >=20 > To me, it looks like an version conflict. Any suggestions how to solve = this? >=20 > Thanks! > Wouter >=20 > [1] - Twitter4s: = https://github.com/DanielaSfregola/twitter4s/blob/master/build.sbt = >=20 --Apple-Mail=_5BA9E806-2EB4-474F-9CF2-8D1308D89760 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Hi,

Please = take a look on this thread first:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble= .com/Akka-Http-used-in-custom-RichSourceFunction-td20314.html

Piotrek

On 11 Jun 2018, at 11:16, Wouter Zorgdrager <zorgdragerw@gmail.com> wrote:

Hi,

I = think I'm running into an Akka version conflict when running a Flink job = on a cluster.

The current situation:
- Flink cluster = on Flink 1.4.2 (using Docker)
- Flink job which = uses twitter4s [1] library and Akka version 2.5.8
In my Flink job I try to 'shutdown' an = Akka actor from the twitter4s library.
This results = in a whole taskmanager crashing with the following stacktrace:

taskrunner_1  | 2018-06-11 09:03:14,454 INFO  = org.apache.flink.runtime.taskmanager.TaskManager      =         - Un-registering task and sending final = execution state CANCELED to JobManager for task Source: Custom Source = -> Sink: Unnamed (0ba7f7f259eee06fe2f7d783c868179b)
taskrunner_1  | Uncaught error from thread = [twitter4s-streaming-akka.actor.default-dispatcher-288]: loader = constraint violation: when resolving method = "akka.actor.ActorCell$$anonfun$3.<init>(Lakka/actor/ActorCell;)V" = the class loader (instance of = org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$= ChildFirstClassLoader) of the current class, akka/actor/ActorCell, and = the class loader (instance of sun/misc/Launcher$AppClassLoader) for the = method's defining class, akka/actor/ActorCell$$anonfun$3, have different = Class objects for the type akka/actor/ActorCell used in the signature, = shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for = for ActorSystem[twitter4s-streaming]
taskrunner_1  | java.lang.LinkageError: loader = constraint violation: when resolving method = "akka.actor.ActorCell$$anonfun$3.<init>(Lakka/actor/ActorCell;)V" = the class loader (instance of = org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$= ChildFirstClassLoader) of the current class, akka/actor/ActorCell, and = the class loader (instance of sun/misc/Launcher$AppClassLoader) for the = method's defining class, akka/actor/ActorCell$$anonfun$3, have different = Class objects for the type akka/actor/ActorCell used in the = signature
taskrunner_1  |    =  at akka.actor.ActorCell.invoke(ActorCell.scala:499)
taskrunner_1  |     at = akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
taskrunner_1  |     at = akka.dispatch.Mailbox.run(Mailbox.scala:224)
taskrunner_1  |     at = akka.dispatch.Mailbox.exec(Mailbox.scala:234)
taskrunner_1  |     at = akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
taskrunner_1  |     at = akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:13= 39)
taskrunner_1  |     at = akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
taskrunner_1  |     at = akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:= 107)
taskrunner_1  | 2018-06-11 09:03:14,984 = INFO  org.apache.flink.runtime.blob.PermanentBlobCache    =           - Shutting down BLOB cache
taskrunner_1  | 2018-06-11 09:03:14,985 INFO  = org.apache.flink.runtime.blob.TransientBlobCache      =         - Shutting down BLOB cache
taskrunner_1  | Exception in thread = "twitter4s-streaming-shutdown-hook-1" java.lang.NoClassDefFoundError: = akka/actor/CoordinatedShutdown$$anonfun$totalTimeout$1
taskrunner_1  |     at = akka.actor.CoordinatedShutdown.totalTimeout(CoordinatedShutdown.scala:515)=
taskrunner_1  |     at = akka.actor.CoordinatedShutdown$$anonfun$initJvmHook$1.apply(CoordinatedShu= tdown.scala:217)
taskrunner_1  |    =  at = akka.actor.CoordinatedShutdown$$anon$2.run(CoordinatedShutdown.scala:547)<= /div>
taskrunner_1  | Caused by: = java.lang.ClassNotFoundException: = akka.actor.CoordinatedShutdown$$anonfun$totalTimeout$1
taskrunner_1  |     at = java.net.URLClassLoader.findClass(URLClassLoader.java:381)
taskrunner_1  |     at = java.lang.ClassLoader.loadClass(ClassLoader.java:424)
taskrunner_1  |     at = org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$= ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)
<= div class=3D"">taskrunner_1  |     at = java.lang.ClassLoader.loadClass(ClassLoader.java:357)
taskrunner_1  |     ... 3 = more

To = me, it looks like an version conflict. Any suggestions how to solve = this?

Thanks!
Wouter

[1] - Twitter4s: https://github.com/DanielaSfregola/twitter4s/blob/master/build.= sbt


= --Apple-Mail=_5BA9E806-2EB4-474F-9CF2-8D1308D89760--