flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Knauf <konstan...@ververica.com>
Subject Re: Starting Flink cluster and running a job
Date Wed, 06 Mar 2019 10:27:15 GMT
Hi Boris,

I was not able tot reproduce your problem. I am using the
docker-entrypoint.sh and Dockerfile from the flink-container project [1].

The content of my fat-jar:

flinkShadowJar
+--- project :fe-flink-common
+--- org.apache.flink:flink-metrics-influxdb_2.11:1.8-SNAPSHOT
+--- org.apache.flink:flink-metrics-slf4j:1.7.2
+--- org.apache.flink:flink-connector-kafka_2.11:1.7.2
|    +--- org.apache.flink:flink-connector-kafka-base_2.11:1.7.2
|    \--- org.apache.kafka:kafka-clients:2.0.1
|         +--- org.lz4:lz4-java:1.4.1
|         \--- org.xerial.snappy:snappy-java:1.1.7.1
\--- org.apache.flink:flink-s3-fs-presto:1.7.2

In the standard Docker image the job.jar is "only" linked into the Flink
lib directory. I also tested it with a modified Dokerfile, which copies the
job.jar into the lib directory of Flink.

bash-4.4$ ls -lia lib
total 135140
 611154 drwxrwxr-x    1 flink    flink         4096 Mar  6 10:20 .
 606092 drwxrwxr-x    1 flink    flink         4096 Feb 11 15:38 ..
26477916 -rw-r--r--    1 flink    flink     93445474 Feb 11 15:38
flink-dist_2.11-1.7.2.jar
26477917 -rw-r--r--    1 flink    flink       141937 Feb 11 15:37
flink-python_2.11-1.7.2.jar
26477525 -rw-r--r--    1 flink    flink     44278062 Mar  6 10:20 job.jar
26477918 -rw-rw-r--    1 flink    flink       489884 Feb 11 14:32
log4j-1.2.17.jar
26477919 -rw-rw-r--    1 flink    flink         9931 Feb 11 14:32
slf4j-log4j12-1.7.15.jar

This also works. In that case you only need to make sure, that the job.jar
is owned by the *flink* user by epxlicitly changing the ownership in the
Dockerfile. Given your description of the Dockerfile, I suspect this is the
problem.

Cheers,

Konstantin

[1] https://github.com/apache/flink/tree/release-1.7.2/flink-container

On Wed, Feb 27, 2019 at 11:19 PM Boris Lublinsky <
boris.lublinsky@lightbend.com> wrote:

> I am not including Kafka per se - I am using Flink Kafka connector, that
> pulls in Kafka.
> I tried exclude and it failed.
>
> I can’t quite shade Kafka and it does not seem like the right thing to do.
> I tried to see where it is loaded from, and its my jar, but I see its
> being loaded twice when it is in Lib.
>
> You guys should be able to easily reproduce it on any Kafka sample that
> you have
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 27, 2019, at 2:10 PM, Ken Krugler <kkrugler_lists@transpac.com>
> wrote:
>
> Hi Boris,
>
> 1. I assume you’ve tried marking the Kafka dependencies as “provided”, and
> that also failed, yes?
>
> 2. Did you try shading the Kafka dependencies?
>
> — Ken
>
> PS - Moving this conversation back to the user list.
>
>
> On Feb 27, 2019, at 2:03 PM, Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
> Guys, please,
> I do still need help with this one
> It does work if I put my jar into a random directory, but fails if I put
> it into Lib.
> And I do need to put it into lib, because standalone-job.sh does not take
> a jar, it assumes a jar to be in lib
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 26, 2019, at 7:33 AM, Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
> Any progress on this one?
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 24, 2019, at 5:16 PM, Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
> And there are more explanations and concerns there
> It turns out that the execution depends on where my jar is located.
> If I put it in lib directory (I need this to be able to run
> bin/standalone-job.sh )
> It fails. With the same error.
>
>
> Currently I have the following in this directory
> fdp-flink-taxiride-assembly-2.0.0.jar  flink-metrics-prometheus-1.7.2.jar
> flink-queryable-state-runtime_2.11-1.7.2.jar  log4j-1.2.17.jar
> flink-dist_2.11-1.7.2.jar       flink-python_2.11-1.7.2.jar
> flink-table_2.11-1.7.2.jar slf4j-log4j12-1.7.15.jar
>
> And no one of them seem to directly contain Kafka (except for my jar
> fdp-flink-taxiride-assembly-2.0.0.jar)
>
> But id I move my jar in any other directory it works fine.
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 23, 2019, at 11:40 AM, Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
> And its working now.
> Sorry about all confusion.
> The root cause of the problem was that in “standard” Link image directory
> /opt/flink/log does not have write permissions and as a result the local
> cluster was never created
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <dwysakowicz@apache.org>
> wrote:
>
> Hi,
>
> One additional question, how do you actually build the docker image? How
> do you put the user jar into the container? Maybe you added extra kafka
> connector to the cluster classpath? Have you checked what is on the
> classpath of a running taskmanager?
>
> Best,
>
> Dawid
> On 22/02/2019 15:44, Boris Lublinsky wrote:
>
> Gordon, I double checked it several times. Here is the list
>
> Columns are
> [info]  - Jar-Size including dependencies
> [info]  - Jar-Size
> [info]  - Number of transitive dependencies
> [info]  - Number of direct dependencies
> [info]  - ModuleID
> [info] Done updating.
> [info]    TotSize    JarSize #TDe #Dep Module
> [info]  27.988 MB ------- MB   33    2
> lightbend:fdp-flink-taxiride_2.11:2.0.0
> [info]  19.632 MB ------- MB   24    6 lightbend:support_2.11:2.0.0
> [info]  17.675 MB   0.000 MB   16    9 org.typelevel:cats_2.11:0.9.0
> [info]  17.389 MB   1.182 MB   13    9 org.typelevel:cats-laws_2.11:0.9.0
> [info]  13.115 MB   0.286 MB    7    4 org.typelevel:cats-free_2.11:0.9.0
> [info]  12.830 MB   0.000 MB    7    4 org.typelevel:cats-jvm_2.11:0.9.0
> [info]  12.830 MB   3.403 MB    6    4 org.typelevel:cats-core_2.11:0.9.0
> [info]   8.397 MB   0.068 MB    8    5
> org.apache.flink:flink-connector-kafka_2.11:1.7.1
> [info]   8.175 MB   1.889 MB    4    4 org.apache.kafka:kafka-clients:2.1.0
> [info]   7.973 MB   0.212 MB    7    4
> org.typelevel:cats-kernel-laws_2.11:0.9.0
> [info]   4.834 MB   0.007 MB    4    2 org.typelevel:cats-macros_2.11:0.9.0
> [info]   4.746 MB   0.082 MB    2    2
> com.typesafe.scala-logging:scala-logging_2.11:3.9.0
> [info]   4.658 MB   0.035 MB    1    1 org.typelevel:machinist_2.11:0.6.1
> [info]   4.623 MB   4.623 MB    0    0 org.scala-lang:scala-reflect:2.11.12
> [info]   4.592 MB   4.592 MB    0    0 org.typelevel:cats-kernel_2.11:0.9.0
> [info]   3.714 MB   3.714 MB    0    0 com.github.luben:zstd-jni:1.3.5-4
> [info]   3.152 MB   0.043 MB    2    1 org.typelevel:discipline_2.11:0.7.2
> [info]   3.109 MB   3.094 MB    1    1
> org.scalacheck:scalacheck_2.11:1.13.4
> [info]   2.019 MB   2.019 MB    0    0
> org.xerial.snappy:snappy-java:1.1.7.2
> [info]   0.803 MB   0.290 MB    2    2 ch.qos.logback:logback-classic:1.2.3
> [info]   0.641 MB   0.641 MB    0    0 joda-time:joda-time:2.10.1
> [info]   0.512 MB   0.512 MB    0    0 org.lz4:lz4-java:1.5.0
> [info]   0.472 MB   0.472 MB    0    0 ch.qos.logback:logback-core:1.2.3
> [info]   0.286 MB   0.286 MB    0    0 com.typesafe:config:1.3.3
> [info]   0.195 MB   0.114 MB    3    3
> org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
> [info]   0.170 MB   0.167 MB    1    1
> com.github.mpilquist:simulacrum_2.11:0.10.0
> [info]   0.145 MB   0.145 MB    0    0 org.joda:joda-convert:2.1.2
> [info]   0.041 MB   0.041 MB    0    0 org.slf4j:slf4j-api:1.7.25
> [info]   0.033 MB   0.033 MB    0    0
> com.google.code.findbugs:jsr305:1.3.9
> [info]   0.016 MB   0.002 MB    2    1
> org.typelevel:catalysts-platform_2.11:0.0.5
> [info]   0.015 MB   0.012 MB    1    1
> org.typelevel:catalysts-macros_2.11:0.0.5
> [info]   0.015 MB   0.015 MB    0    0 org.scala-sbt:test-interface:1.0
> [info]   0.007 MB   0.007 MB    0    0 org.apache.flink:force-shading:1.7.1
> [info]   0.003 MB   0.003 MB    0    0
> org.typelevel:macro-compat_2.11:1.1.1
>
>
> And here is another view
>
> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
> [info]   +-lightbend:support_2.11:2.0.0 [S]
> [info]   | +-ch.qos.logback:logback-classic:1.2.3
> [info]   | | +-ch.qos.logback:logback-core:1.2.3
> [info]   | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info]   | | +-org.slf4j:slf4j-api:1.7.25
> [info]   | |
> [info]   | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
> [info]   | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info]   | | +-org.slf4j:slf4j-api:1.7.25
> [info]   | |
> [info]   | +-com.typesafe:config:1.3.3
> [info]   | +-joda-time:joda-time:2.10.1
> [info]   | +-org.joda:joda-convert:2.1.2
> [info]   | +-org.typelevel:cats_2.11:0.9.0 [S]
> [info]   |   +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   |
> [info]   |   +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info]   |   | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]   |   | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]   |   | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   |
> [info]   |   +-org.typelevel:cats-free_2.11:0.9.0 [S]
> [info]   |   | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info]   |   | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]   |   | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]   |   | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | | |
> [info]   |   | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   | | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]   |   | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   |
> [info]   |   +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
> [info]   |   | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info]   |   | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]   |   | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]   |   | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | | |
> [info]   |   | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   | | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]   |   | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   |
> [info]   |   +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info]   |   | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]   |   | | +-org.scala-sbt:test-interface:1.0
> [info]   |   | |
> [info]   |   | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info]   |   | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info]   |   | |   +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]   |   | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info]   |   |   +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]   |   |     +-org.scala-sbt:test-interface:1.0
> [info]   |   |
> [info]   |   +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]   |   +-org.typelevel:cats-laws_2.11:0.9.0 [S]
> [info]   |   | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | |
> [info]   |   | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]   |   | | +-org.scala-sbt:test-interface:1.0
> [info]   |   | |
> [info]   |   | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info]   |   | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info]   |   | |   +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info]   |   | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]   |   | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]   |   | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | | |
> [info]   |   | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   | | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info]   |   | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]   |   | | | +-org.scala-sbt:test-interface:1.0
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info]   |   | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info]   |   | | |   +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]   |   | | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info]   |   | |   +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]   |   | |     +-org.scala-sbt:test-interface:1.0
> [info]   |   | |
> [info]   |   | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]   |   | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]   |   | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | | |
> [info]   |   | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info]   |   | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]   |   | |   +-org.scala-sbt:test-interface:1.0
> [info]   |   | |
> [info]   |   | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   |
> [info]   |   +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]   |   | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]   |   | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]   |   | |
> [info]   |   | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |   |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |   |
> [info]   |   +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]   |     +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   |
> [info]   +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
> [info]     +-com.google.code.findbugs:jsr305:1.3.9
> [info]     +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
> [info]     | +-com.google.code.findbugs:jsr305:1.3.9
> [info]     | +-org.apache.flink:force-shading:1.7.1
> [info]     | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info]     | +-org.slf4j:slf4j-api:1.7.25
> [info]     |
> [info]     +-org.apache.flink:force-shading:1.7.1
> [info]     +-org.apache.kafka:kafka-clients:2.1.0
> [info]     | +-com.github.luben:zstd-jni:1.3.5-4
> [info]     | +-org.lz4:lz4-java:1.5.0
> [info]     | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info]     | +-org.slf4j:slf4j-api:1.7.25
> [info]     | +-org.xerial.snappy:snappy-java:1.1.7.2
> [info]     |
> [info]     +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info]     +-org.slf4j:slf4j-api:1.7.25
> [info]
> [info] Done updating.
> [info] lightbend:support_2.11:2.0.0 [S]
> [info]   +-ch.qos.logback:logback-classic:1.2.3
> [info]   | +-ch.qos.logback:logback-core:1.2.3
> [info]   | +-org.slf4j:slf4j-api:1.7.25
> [info]   |
> [info]   +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
> [info]   | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]   | +-org.slf4j:slf4j-api:1.7.25
> [info]   |
> [info]   +-com.typesafe:config:1.3.3
> [info]   +-joda-time:joda-time:2.10.1
> [info]   +-org.joda:joda-convert:2.1.2
> [info]   +-org.typelevel:cats_2.11:0.9.0 [S]
> [info]     +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     |
> [info]     +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info]     | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | |
> [info]     | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]     | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]     | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     | |
> [info]     | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     |
> [info]     +-org.typelevel:cats-free_2.11:0.9.0 [S]
> [info]     | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | |
> [info]     | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info]     | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]     | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]     | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | | |
> [info]     | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     | | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     | |
> [info]     | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]     | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     | |
> [info]     | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     |
> [info]     +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
> [info]     | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | |
> [info]     | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info]     | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]     | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]     | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | | |
> [info]     | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     | | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     | |
> [info]     | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]     | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     | |
> [info]     | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     |
> [info]     +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info]     | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]     | | +-org.scala-sbt:test-interface:1.0
> [info]     | |
> [info]     | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info]     | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info]     | |   +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | |
> [info]     | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]     | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info]     |   +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]     |     +-org.scala-sbt:test-interface:1.0
> [info]     |
> [info]     +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]     +-org.typelevel:cats-laws_2.11:0.9.0 [S]
> [info]     | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | |
> [info]     | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]     | | +-org.scala-sbt:test-interface:1.0
> [info]     | |
> [info]     | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info]     | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info]     | |   +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | |
> [info]     | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info]     | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]     | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]     | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | | |
> [info]     | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     | | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     | |
> [info]     | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info]     | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]     | | | +-org.scala-sbt:test-interface:1.0
> [info]     | | |
> [info]     | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info]     | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info]     | | |   +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]     | | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info]     | |   +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]     | |     +-org.scala-sbt:test-interface:1.0
> [info]     | |
> [info]     | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info]     | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]     | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | | |
> [info]     | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     | |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     | |
> [info]     | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info]     | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info]     | |   +-org.scala-sbt:test-interface:1.0
> [info]     | |
> [info]     | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     |
> [info]     +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info]     | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info]     | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info]     | |
> [info]     | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]     |   +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]     |
> [info]     +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info]       +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]
>
>
>
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <gordon@ververica.com>
> wrote:
>
> Hi,
>
> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a
> reoccurrence of FLINK-8741.
> I've double checked the used classloaders in the Kafka connector, they
> seem to be correct.
>
> The fact that it works correctly in IntelliJ, but not when packaged,
> suggests that there could be some conflicting dependencies in the packaged
> jar.
>
> Could you check the actual resolved dependency tree of the project, and
> see if there are multiple versions of some dependency related to Kafka
> being pulled in? For Maven for example, that would be "mvn dependency:tree".
>
> Cheers,
> Gordon
>
> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
>> The relevant dependencies are
>>
>> val flinkScala            =      "org.apache.flink"             %%   "flink-scala"
                   % flinkVersion % "provided"val flinkStreamingScala   =      "org.apache.flink"
            %%   "flink-streaming-scala"          % flinkVersion % "provided"val flinkKafka
           =      "org.apache.flink"             %%   "flink-connector-kafka"          % flinkVersion
exclude("org.slf4j", "slf4j-log4j12")
>>
>> I am using SBT
>> I tried both connector-kafka and connector-kaka-11 - same result
>>
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>>
>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <konstantin@ververica.com>
>> wrote:
>>
>> Hi Boris,
>>
>> can you the relevant parts (dependencies) of your pom.xml? Did you also
>> try without fixing the Kafka version, i.e. running with the Kafka client
>> version provided by the Kafka connector of Flink? Gordon (cc) dealt with
>> FLINK-8741.
>>
>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>
>> Cheers,
>>
>> Konstantin
>>
>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <
>> boris.lublinsky@lightbend.com> wrote:
>>
>>> I found some more details on this
>>> The same error for the same application was reported about a year ago
>>> http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E
>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741
>>>
>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with
>>> both latest kaffka-connector
>>> And Kafka-connector-011
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com
>>> https://www.lightbend.com/
>>>
>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kkrugler_lists@transpac.com>
>>> wrote:
>>>
>>> Hi Boris,
>>>
>>> I haven’t seen this exact error, but I have seen similar errors caused
>>> by multiple versions of jars on the classpath.
>>>
>>> When I’ve run into this particular "XXX is not an instance of YYY"
>>> problem, it often seems to be caused by a jar that I should have marked as
>>> provided in my pom.
>>>
>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this
>>> doesn’t apply.
>>>
>>> — Ken
>>>
>>> PS - I assume you’ve been reading
>>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html
>>>
>>>
>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <
>>> boris.lublinsky@lightbend.com> wrote:
>>>
>>> Konstantin,
>>> After experimenting with this for a while, I got to the root cause of
>>> the problem
>>> I am running a version of a Taxi ride travel prediction as my sample.
>>> It works fine in Intellij,
>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>> It fails with a following error
>>>
>>>
>>> The program finished with the following exception:
>>>
>>> org.apache.flink.client.program.ProgramInvocationException: Job failed.
>>> (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>> at
>>> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>> at
>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>> at
>>> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>> at
>>> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>> at
>>> com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>> at
>>> com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at
>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>> at
>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>> at
>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>> at
>>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
>>> execution failed.
>>> at
>>> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>> at
>>> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>> ... 19 more
>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct
>>> kafka producer
>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>> at
>>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>> at
>>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>> at
>>> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>> at
>>> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>> at
>>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>> at
>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.kafka.common.KafkaException:
>>> org.apache.kafka.common.serialization.ByteArraySerializer is not an
>>> instance of org.apache.kafka.common.serialization.Serializer
>>> at
>>> org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>> ... 17 more
>>>
>>> The closest that I found
>>> https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no
>>> Which talks about class loader. (I tried there solution, but it did not
>>> help)
>>> I looked at the loading and I see that the pair of these 2 classes is
>>> loaded from my uber jar, but twice.
>>>
>>> Have you guys seen this error before?
>>> Any suggestion?
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com
>>> https://www.lightbend.com/
>>>
>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com>
>>> wrote:
>>>
>>> Hi Boris,
>>>
>>> without looking at the entrypoint in much detail, generally there should
>>> not be a race condition there:
>>>
>>> * if the taskmanagers can not connect to the resourcemanager they will
>>> retry (per default the timeout is 5 mins)
>>> * if the JobManager does not get enough resources from the
>>> ResourceManager it will also wait for the resources/slots to provided. The
>>> timeout there is also 5 minutes, I think.
>>>
>>> So, this should actually be pretty robust as long as the Taskmanager
>>> containers can reach the Jobmanager eventually.
>>>
>>> Could you provide the Taskmanager/JobManager logs for such a failure
>>> case?
>>>
>>> Cheers,
>>>
>>> Konstantin
>>>
>>>
>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <
>>> boris.lublinsky@lightbend.com> wrote:
>>>
>>>> Following
>>>> https://github.com/apache/flink/tree/release-1.7/flink-container/docker
>>>> I have created an entry point, which looks like follows:
>>>>
>>>> #!/bin/sh#################################################################################
  from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh#
  and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh#################################################################################
If unspecified, the hostname of the container is taken as the JobManager addressJOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname
-f)}
>>>> drop_privs_cmd() {
>>>>     if [ $(id -u) != 0 ]; then        # Don't need to drop privs if EUID
!= 0        return    elif [ -x /sbin/su-exec ]; then        # Alpine        echo su-exec
flink
>>>>     else        # Others        echo gosu flink
>>>>     fi}
>>>>
>>>> JOB_MANAGER="jobmanager"TASK_MANAGER="taskmanager"CMD="$1"shiftif [ "${CMD}"
= "help" ]; then    echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"   
exit 0elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then    if [ "${CMD}"
= "${TASK_MANAGER}" ]; then        TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep
-c ^processor /proc/cpuinfo)}
>>>>
>>>>         sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address:
${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"        sed -i -e "s/taskmanager.numberOfTaskSlots:
1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
       echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"        echo
"query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"        echo "Starting
Task Manager"        echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
       exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>     else        sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address:
${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"        echo "blob.server.port:
6124" >> "$FLINK_HOME/conf/flink-conf.yaml"        echo "query.server.port: 6125" >>
"$FLINK_HOME/conf/flink-conf.yaml"        echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
       if [ -z "$1" ]; then           exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh"
start-foreground "$@"        else            exec $FLINK_HOME/bin/standalone-job.sh start-foreground
"$@"        fi    fifiexec "$@"
>>>>
>>>> It does work for all the cases, except running standalone job.
>>>> The problem, the way I understand it, is a racing condition.
>>>> In kubernetes it takes several attempts for establish connection
>>>> between Job and Task manager, while standalone-job.sh
>>>>  tries to start a job immediately once the cluster is created (before
>>>> connection is established).
>>>> Is there a better option to implement it starting a job on container
>>>> startup?
>>>>
>>>>
>>>
>>>
>>> --
>>> Konstantin Knauf | Solutions Architect
>>> +49 160 91394525
>>>
>>>
>>> --------------------------
>>> Ken Krugler
>>> +1 530-210-6378
>>> http://www.scaleunlimited.com
>>> Custom big data solutions & training
>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>
>>>
>>>
>>
>> --
>> Konstantin Knauf | Solutions Architect
>> +49 160 91394525
>> <https://www.ververica.com/>
>>
>> Follow us @VervericaData
>> --
>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> Conference
>> Stream Processing | Event Driven | Real Time
>> --
>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>> --
>> Data Artisans GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing
>> Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>
>>
>>
>
>
>
>
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
>
>
>

-- 

Konstantin Knauf | Solutions Architect

+49 160 91394525

<https://www.ververica.com/>

Follow us @VervericaData

--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

Mime
View raw message