Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CDDF718360 for ; Thu, 14 Jan 2016 01:26:54 +0000 (UTC) Received: (qmail 45748 invoked by uid 500); 14 Jan 2016 01:26:49 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 45651 invoked by uid 500); 14 Jan 2016 01:26:49 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 45641 invoked by uid 99); 14 Jan 2016 01:26:49 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2016 01:26:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1E062C1F06 for ; Thu, 14 Jan 2016 01:26:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.902 X-Spam-Level: ** X-Spam-Status: No, score=2.902 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=correlatesystems.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id X5L5tyFy8L-s for ; Thu, 14 Jan 2016 01:26:37 +0000 (UTC) Received: from mail-qk0-f177.google.com (mail-qk0-f177.google.com [209.85.220.177]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 6CA1031ACF for ; Thu, 14 Jan 2016 01:26:36 +0000 (UTC) Received: by mail-qk0-f177.google.com with SMTP id y67so11224378qkc.2 for ; Wed, 13 Jan 2016 17:26:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=correlatesystems.com; s=google; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to; bh=TupsuTvtneCCux+03kU1utIDdF2ljUYeCjwkQrn7TBg=; b=AUznynnoiXyqt6kLOdrrZJo2TIXVslo3wNvhIWTeGAxZiu43vkiEn3KgC3ljF9Nl1Z dd2MuGAebybavySCG1hF7P+ShppXu5ans+oarFKmUPBAhsKg9PmPRGNWG12tccT85dGf yvhRNwmuPk5gnXNFK8VSJgIJH6EOCdfGGI56A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=TupsuTvtneCCux+03kU1utIDdF2ljUYeCjwkQrn7TBg=; b=LyOk6BH58KFI+MssX5bIjf3G1wYMrHPHA3omCWkTk/1xac7ad6T/upU2anMtgvEgXh MdyonzJ+Q0yu8YEAbYyfncWfyzOu8AkoJCHYLEtVagSaZFOs94U/yL1mOr9RVo9W9uYB Ssx7Q/we5nw4xFKP+pL1PCgpD5VCKpBy3TnMyTgp7eljxeNqkyUyp71HcrBQe4/QDwgQ KT/GsCwtvbxiys3h+EyyRAogTOQm3G27NpmgEQX4EQd+dEDvA9Rxh5LJPD1RR/DqWJCi W6ut0KBp2/ETLkYCdIBljVhR2dKY9bpuCMPv72QAwyHw/i5AMKU2TEdUtJgQncy0MUQt aU2w== X-Gm-Message-State: ALoCoQletjeZlEFMxF3WdY7WyB/paftwmasBx0RfgFvvwcLY7PGa8z90exQS3YvtrNpNhuYLWwATV9l6mWOLlgs7V1hrKWZSkw== X-Received: by 10.55.22.9 with SMTP id g9mr1666268qkh.12.1452734795055; Wed, 13 Jan 2016 17:26:35 -0800 (PST) Received: from [10.0.40.101] (static-100-0-198-149.bstnma.fios.verizon.net. [100.0.198.149]) by smtp.gmail.com with ESMTPSA id n35sm1604803qgn.10.2016.01.13.17.26.33 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 13 Jan 2016 17:26:33 -0800 (PST) From: Prez Cannady Content-Type: multipart/alternative; boundary="Apple-Mail=_232A76CF-2DDE-4D34-9A66-D609F5318105" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: Flink QuickStart: On start up, running into ClassNotFoundException: org.apache.flink.streaming.runtime.tasks.OneInputStreamTask Date: Wed, 13 Jan 2016 20:26:31 -0500 References: <84861085-942E-416B-A608-F9E3D7426D30@opencorrelate.org> To: user@flink.apache.org In-Reply-To: X-Mailer: Apple Mail (2.3112) --Apple-Mail=_232A76CF-2DDE-4D34-9A66-D609F5318105 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Simply passing FlinkUserCodeClassLoader.class.getClassLoader to the = parent constructor cleared the impasse.=20 2016-01-13 20:06:43.637 INFO 35403 --- [ main] = o.o.e.j.s.SocketTextStreamWordCount$ : Started = SocketTextStreamWordCount. in 5.176 seconds (JVM running for 12.58) [INFO] = ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] = ------------------------------------------------------------------------ [INFO] Total time: 11.734 s [INFO] Finished at: 2016-01-13T20:06:43-05:00 [INFO] Final Memory: 49M/4986M [INFO] = ------------------------------------------------------------------------ 2016-01-13 20:06:43.804 INFO 35403 --- [ Thread-3] = s.c.a.AnnotationConfigApplicationContext : Closing = org.springframework.context.annotation.AnnotationConfigApplicationContext@= 33248c18: startup date [Wed Jan 13 20:06:38 EST 2016]; root of context = hierarchy 2016-01-13 20:06:43.806 INFO 35403 --- [ Thread-3] = o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed = beans on shutdown All tests in flink-runtime passed after the change = `BlobLibraryCacheManager=E2=80=99, but I haven=E2=80=99t run the full = test suite.=20 Is this actually an appropriate fix, or just a way to highlight a = configuration problem?=20 I assume that injecting a parent class loader when registering a task = might break things, but I don=E2=80=99t know nearly enough about Flink = and this code to say one way or another. Prez Cannady =20 p: 617 500 3378 =20 e: revprez@opencorrelate.org =20 GH: https://github.com/opencorrelate =20= LI: https://www.linkedin.com/in/revprez = =20 > On Jan 13, 2016, at 6:50 PM, Stephan Ewen wrote: >=20 > Hi! >=20 > Running this is Spring, the whole classloader configuration is = probably a bit different than in Flink's standalone or YARN or local = mode. >=20 > Can you try if the following solves your problem:=20 >=20 > At the end of the file "BlobLibraryCacheManager", there is the private = class "FlinkUserCodeClassloader". >=20 > Can you replace the current FlinkUserCodeClassloader with this? >=20 >=20 > private static class FlinkUserCodeClassLoader extends URLClassLoader { >=20 > public FlinkUserCodeClassLoader(URL[] urls) { > super(urls, = FlinkUserCodeClassLoader.class.getClassLoader()); > } > } >=20 > You can also try and use instead of = "FlinkUserCodeClassLoader.class.getClassLoader()" the statements = "Thread.currentThread().getContextClassLoader()". >=20 > Let me know if one of the two solves the problem. >=20 > Greetings, > Stephan >=20 >=20 > On Wed, Jan 13, 2016 at 7:20 PM, Prez Cannady = > wrote: > I=E2=80=99m experimenting combining Spring with Flink. I=E2=80=99ve = successfully instrumented for Gradle, but Maven is emitting = ClassNotFoundExceptions for items ostensibly on the class path. >=20 > Project is currently configured for: >=20 > 1. Scala 2.10.4 > 2. Flink 0.9.1 >=20 > I execute the following >=20 > ``` > # In one terminal > $ nc -lk -p 9999 --sh-exec "cat /usr/share/dict/words | head -n 10=E2=80= =9D >=20 >=20 > # In another terminal > $ mvn clean install spring-boot:run = -Drun.arguments=3D=E2=80=9Clocalhost,9999=E2=80=9D >=20 > # observe output > ``` >=20 > The specific class not found is = org.apache.flink.streaming.runtime.tasks.OneInputStreamTask. However, = Spring Boot Plugin is configured to repackage a fat jar, and I can see = that the class is present in the included flink-streaming-core jar. = Additionally, LogBack shows that the flink-streaming-core jar is in my = classpath. >=20 >=20 > I=E2=80=99m hoping I=E2=80=99m just missing something that should be = obvious. While I wish could move forward with just Gradle, = unfortunately I have to support Maven builds. >=20 > For reference, the complete project is available here: >=20 > https://github.com/OCExercise/wordcount-processing = >=20 > Additionally >=20 > 1. pom.xml = (https://github.com/OCExercise/wordcount-processing/blob/master/pom.xml = ) > 2. build.grade = (https://github.com/OCExercise/wordcount-processing/blob/master/build.grad= le = ) > 3. Gist containing the full exception = (https://gist.github.com/revprez/2c1fb01c40e5d6790247 = ) >=20 > Prez Cannady =20 > p: 617 500 3378 =20 > e: revprez@opencorrelate.org =20 > GH: https://github.com/opencorrelate = =20 > LI: https://www.linkedin.com/in/revprez = =20 >=20 >=20 --Apple-Mail=_232A76CF-2DDE-4D34-9A66-D609F5318105 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

Simply passing FlinkUserCodeClassLoader.class.getClassLoader to the = parent constructor cleared the impasse. 

2016-01-13 = 20:06:43.637  INFO 35403 --- [           = main] o.o.e.j.s.SocketTextStreamWordCount$     : Started = SocketTextStreamWordCount. in 5.176 seconds (JVM running for 12.58)

[INFO] = ------------------------------------------------------------------------

[INFO] BUILD = SUCCESS

[INFO] = ------------------------------------------------------------------------

[INFO] Total = time: 11.734 s

[INFO] = Finished at: 2016-01-13T20:06:43-05:00

[INFO] Final = Memory: 49M/4986M

[INFO] = ------------------------------------------------------------------------

2016-01-13 = 20:06:43.804  INFO 35403 --- [       Thread-3] = s.c.a.AnnotationConfigApplicationContext : Closing = org.springframework.context.annotation.AnnotationConfigApplicationContext@= 33248c18: startup date [Wed Jan 13 20:06:38 EST 2016]; root of context = hierarchy

2016-01-13 = 20:06:43.806  INFO 35403 --- [       Thread-3] = o.s.j.e.a.AnnotationMBeanExporter        : = Unregistering JMX-exposed beans on shutdown


All tests in flink-runtime passed = after the change `BlobLibraryCacheManager=E2=80=99, but I haven=E2=80=99t = run the full test suite. 

Is this actually an appropriate fix, or just a way to = highlight a configuration problem? 

I assume that injecting a parent class loader when = registering a task might break things, but I don=E2=80=99t know nearly = enough about Flink and this code to say one way or = another.



=





On Jan 13, 2016, at 6:50 PM, Stephan Ewen <sewen@apache.org> = wrote:

Hi!

Running this is Spring, the whole classloader configuration = is probably a bit different than in Flink's standalone or YARN or local = mode.

Can you = try if the following solves your problem: 

At the end of the file = "BlobLibraryCacheManager", there is the private class = "FlinkUserCodeClassloader".

Can you replace the current FlinkUserCodeClassloader with = this?


private static class FlinkUserCodeClassLoader = extends URLClassLoader {

public = FlinkUserCodeClassLoader(URL[] urls) {
= super(urls, FlinkUserCodeClassLoader.class.getClassLoader());
= }
}

You can also try and use instead of "FlinkUserCodeClassLoader.class.getClassLoader()" th= e statements "Thread.currentThread().getContextClassLoader()".

Let me know if one of = the two solves the problem.

Greetings,
Stephan


On Wed, Jan 13, 2016 at 7:20 PM, = Prez Cannady <revprez@opencorrelate.org> wrote:
I=E2=80=99m = experimenting combining Spring with Flink.  I=E2=80=99ve = successfully instrumented for Gradle, but Maven is emitting = ClassNotFoundExceptions for items ostensibly on the class = path.

Project = is currently configured for:

1. Scala 2.10.4
2. = Flink 0.9.1

I = execute the following

```
# In one terminal
$ nc -lk -p 9999 --sh-exec "cat /usr/share/dict/words | head = -n 10=E2=80=9D


# In another = terminal
$ mvn clean install spring-boot:run = -Drun.arguments=3D=E2=80=9Clocalhost,9999=E2=80=9D

# observe = output
```

The specific class not found is org.apache.flink.streaming.runtime.tasks.OneInputStreamTask= .  However, Spring Boot Plugin is configured to repackage a fat = jar, and I can see that the class is present in the included = flink-streaming-core jar.  Additionally, LogBack shows that the = flink-streaming-core jar is in my classpath.


I=E2=80=99m hoping I=E2=80=99m just missing something that = should be obvious.  While I wish could move forward with just = Gradle, unfortunately I have to support Maven builds.

For reference, the = complete project is available here:

https://github.com/OCExercise/wordcount-processing

Additionally

1. pom.xml (https://github.com/OCExercise/wordcount-processing/blob/master/= pom.xml)
2. build.grade (https://github.com/OCExercise/wordcount-processing/blob/master/= build.gradle)
3. Gist containing the full = exception (https://gist.github.com/revprez/2c1fb01c40e5d6790247)
=

Prez Cannady  
p: 617 500 3378  
e: revprez@opencorrelate.org  
GH: https://github.com/opencorrelate  
LI: https://www.linkedin.com/in/revprez  



= --Apple-Mail=_232A76CF-2DDE-4D34-9A66-D609F5318105--