flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navneeth Krishnan <reachnavnee...@gmail.com>
Subject Re: Flink on EMR
Date Wed, 27 Sep 2017 04:57:47 GMT
Hi All,

Any suggestions?

Thanks.

On Mon, Sep 25, 2017 at 10:14 PM, Navneeth Krishnan <
reachnavneeth2@gmail.com> wrote:

> Hello All,
>
> I'm trying to deploy flink on AWS EMR and I'm very new to EMR. I'm running
> into multiple issues and need some help.
>
> *Issue1:*
>
> How did others resolve this multiple bindings issue?
>
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1505848894978_0007/filecache/11/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1505848894978_0007/filecache/12/location-compute-1.0-SNAPSHOT-all.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
>
> *Issue2:*
>
> Running the below command runs the pipeline but the task manager is allocated with only
5GB memory instead of 8GB memory. Any reason why?
> flink run -m yarn-cluster -yn 4 -yjm 2048 -ytm 8192 ./my-pipeline.jar
>
>
> *Issue3:*
>
> How to provide the checkpoint directory? By just providing this "hdfs:///checkpoints/"
will it work or should I provide any master node host name?
>
>
> *Issue 4:*
>
> How can I get the task manager logs? Should I use log aggregation in hadoop yarn or send
it to cloud watch?
>
>
> Also if there any best practices to be used while running flink on yarn, please let me
know.
>
>
> Thanks a lot.
>
>
> Regards,
>
> Navneeth
>
>

Mime
View raw message