flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie Grier <ja...@data-artisans.com>
Subject Re: Error submitting stand-alone Flink job to EMR YARN cluster
Date Fri, 01 Jul 2016 23:21:46 GMT
I know this is really basic but have you verified that you're Flink lib
folder contains log4j-1.2.17.jar?  I imagine that's fine given the
yarn-session.sh approach is working fine.  What version of EMR are you
running?  What version of Flink?

-Jamie


On Thu, Jun 30, 2016 at 11:28 AM, Hanson, Bruce <bruce.hanson@here.com>
wrote:

> I’m trying to submit a stand-alone Flink job to a YARN cluster running on
> EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container
> for the Job Manager, it fails. The error message from the container is
> below. The command I’m using is:
>
> $ flink run -m yarn-cluster -yn 1 -ynm test1:WordCount
> ./flink-1.0.3/examples/streaming/WordCount.jar
>
> I have tried adding log4j and slf4j libraries to the classpath using -C
> and that doesn’t help.
>
> This does not happen on other YARN clusters I have that are not EMR nodes.
> And it doesn’t happen on my EMR cluster if I use "yarn-session.sh" to
> create a Flink cluster in the YARN cluster and then use “flink run …” to
> submit the job to the Flink cluster.
>
> Does anyone out there know how I could fix this?
>
> Thanks in advance for any help you can give.
>
> Error message in the jobmanager.err file:
>
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>
> SLF4J: Defaulting to no-operation (NOP) logger implementation
>
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/log4j/Level
>
>         at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:357)
>
>         at java.lang.Class.forName0(Native Method)
>
>         at java.lang.Class.forName(Class.java:278)
>
>         at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
>
>         at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95)
>
>         at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
>
>         at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
>
>         at org.apache.hadoop.security.Groups.<init>(Groups.java:79)
>
>         at org.apache.hadoop.security.Groups.<init>(Groups.java:74)
>
>         at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
>
>         at
> org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64)
>
>         at
> org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36)
>
>         at
> org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala)
>
> Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
>         ... 18 more
>
>
> *Bruce Hanson*
>
> Software Engineer
>
> HERE Predictive Analytics
>
>
>
> *HERE Seattle*
>
> 701 Pike St, Suite 2000, Seattle, WA 98101
>
> *47° 36' 41" N. 122° 19' 57" W
> <http://here.com/usa/seattle/98101/pike-st/701/map=47.611439,-122.332741,17/title=HERE%20Seattle%20-%20701%20Pike%20Street>*
>
>
>
> <http://360.here.com/>   <https://twitter.com/here>
> <https://www.facebook.com/here>    <https://linkedin.com/company/heremaps>
>   <https://www.instagram.com/here>
>
>
>
>


-- 

Jamie Grier
data Artisans, Director of Applications Engineering
@jamiegrier <https://twitter.com/jamiegrier>
jamie@data-artisans.com

Mime
View raw message