Hi, 

I’m using the default flink package that comes with EMR. I’m facing the issue while running my pipeline. Thanks.

On Mon, Sep 25, 2017 at 11:09 PM Jörn Franke <jornfranke@gmail.com> wrote:
Amazon EMR has already a Flink package. You just need to check the checkbox. I would not install it on your own. 
I think you can find it in the advanced options.

On 26. Sep 2017, at 07:14, Navneeth Krishnan <reachnavneeth2@gmail.com> wrote:

Hello All,

I'm trying to deploy flink on AWS EMR and I'm very new to EMR. I'm running into multiple issues and need some help.

Issue1:
How did others resolve this multiple bindings issue?

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1505848894978_0007/filecache/11/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1505848894978_0007/filecache/12/location-compute-1.0-SNAPSHOT-all.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Issue2:
Running the below command runs the pipeline but the task manager is allocated with only 5GB memory instead of 8GB memory. Any reason why?
flink run -m yarn-cluster -yn 4 -yjm 2048 -ytm 8192 ./my-pipeline.jar

Issue3:
How to provide the checkpoint directory? By just providing this "hdfs:///checkpoints/" will it work or should I provide any master node host name?

Issue 4:
How can I get the task manager logs? Should I use log aggregation in hadoop yarn or send it to cloud watch?

Also if there any best practices to be used while running flink on yarn, please let me know.

Thanks a lot.

Regards,
Navneeth