flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chiwan Park <chiwanp...@apache.org>
Subject Flink on EMR Question
Date Mon, 04 Jan 2016 11:52:03 GMT
Hi All,

I have some problems using Flink on Amazon EMR cluster.

Q1. Sometimes, jobmanager container still exists after destroying yarn session by pressing
Ctrl+C. In that case, Flink YARN app seems exited correctly in YARN RM dashboard. But there
is a running container in the dashboard. From logs of the container, I realize that the container
is jobmanager.

I cannot kill the container because there is no permission to restart YARN RM in Amazon EMR.
In my small Hadoop Cluster (w/3 nodes), the problem doesn’t appear.

Q2. I tried to use S3 file system in Flink on EMR. But I can’t use it because of version
conflict of Apache Httpclient. In default, implementation of S3 file system in EMR is `com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem`
which is linked with other version of Apache Httpclient.

As I wrote above, I cannot restart Hadoop cluster after modifying conf-site.xml because of
lack of permission. How can I solve this problem?

Chiwan Park

View raw message