flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jared Stehler <jared.steh...@intellifylearning.com>
Subject Re: hadoop error with flink mesos on startup
Date Tue, 12 Dec 2017 19:31:47 GMT
The class is there; this issue is a static initializer error, probably from other missing classes.
I’ll try using the uber jar to see if that helps any, and will report back.

I’ve included the shaded jar as a maven dependency:


Jared Stehler
Chief Architect - Intellify Learning
o: 617.701.6330 x703

> On Dec 12, 2017, at 2:10 PM, Chesnay Schepler <chesnay@apache.org> wrote:
> Could you look into the flink-shaded-hadoop jar to check whether the missing class is
actually contained?
> Where did the flink-shaded-hadoop jar come from? I'm asking because when building flink-dist
from source the jar is called flink-shaded-hadoop2-uber-1.4.0.jar, which does indeed contain
the jar. (the uber jar is created by building flink-shaded-hadoop2-uber)
> On 12.12.2017 19:28, Jared Stehler wrote:
>> After upgrading to flink 1.4.0 using the hadoop-free build option, I’m seeing the
following error on startup in the app master:
>> 2017-12-12 18:23:15.473 [main] ERROR o.a.f.m.r.clusterframework.MesosApplicationMasterRunner
 - Mesos JobManager initialization failed
>>  <https://internal.dev3.master.us-west-2.prod.aws.intellify.io/singularity/task/flink-appmaster-1513102980-1513102985422-1->java.lang.NoClassDefFoundError:
Could not initialize class org.apache.hadoop.security.UserGroupInformation
>>  <https://internal.dev3.master.us-west-2.prod.aws.intellify.io/singularity/task/flink-appmaster-1513102980-1513102985422-1->
at org.apache.flink.runtime.clusterframework.overlays.HadoopUserOverlay$Builder.fromEnvironment(HadoopUserOverlay.java:74)
>>  <https://internal.dev3.master.us-west-2.prod.aws.intellify.io/singularity/task/flink-appmaster-1513102980-1513102985422-1->
at org.apache.flink.mesos.entrypoint.MesosEntrypointUtils.applyOverlays(MesosEntrypointUtils.java:145)
>> Looking at the code, it appears that the HadoopUserOverlay always tries to init the
UserGroupInformation class, and is failing. Same error with or without the flink-shaded-hadoop2
library included.
>> This is my lib dir:
>> flink-appmaster-1.0-SNAPSHOT.jar   flink-s3-fs-presto-1.4.0.jar       jul-to-slf4j-1.7.25.jar
>> flink-dist_2.11-1.4.0.jar          flink-shaded-hadoop2-1.4.0.jar     log4j-over-slf4j-1.7.25.jar
>> flink-metrics-prometheus-1.4.0.jar jackson-core-2.8.10.jar            logback-classic-1.1.11.jar
>> flink-python_2.11-1.4.0.jar        jcl-over-slf4j-1.7.25.jar          logback-core-1.1.11.jar
>> --
>> Jared Stehler
>> Chief Architect - Intellify Learning
>> o: 617.701.6330 x703

View raw message