flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Bode <maximilian.b...@tngtech.com>
Subject Re: Flink on EMR Question
Date Tue, 05 Jan 2016 13:49:52 GMT
Hi everyone,

Regarding Q1, I believe I have witnessed a comparable phenomenon in a (3-node, non-EMR) YARN
cluster. After shutting down the yarn session via `stop`, one container seems to linger around.
`yarn application -list` is empty, whereas `bin/yarn-session.sh -q` lists the left-over container.
Also, there is still one application shown as ‚running‘ in Ambari’s YARN pane under
current applications. Then, after some time (order of a few minutes) it disappears and the
resources are available again.

I have not tested this behavior extensibly so far. Noticeably, I was not able to reproduce
it by just starting a session and then ending it again right away without looking at the JobManager
web interface. Maybe this produces some kind of lag as far as YARN containers are concerned?


> Am 04.01.2016 um 12:52 schrieb Chiwan Park <chiwanpark@apache.org>:
> Hi All,
> I have some problems using Flink on Amazon EMR cluster.
> Q1. Sometimes, jobmanager container still exists after destroying yarn session by pressing
Ctrl+C. In that case, Flink YARN app seems exited correctly in YARN RM dashboard. But there
is a running container in the dashboard. From logs of the container, I realize that the container
is jobmanager.
> I cannot kill the container because there is no permission to restart YARN RM in Amazon
EMR. In my small Hadoop Cluster (w/3 nodes), the problem doesn’t appear.
> Q2. I tried to use S3 file system in Flink on EMR. But I can’t use it because of version
conflict of Apache Httpclient. In default, implementation of S3 file system in EMR is `com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem`
which is linked with other version of Apache Httpclient.
> As I wrote above, I cannot restart Hadoop cluster after modifying conf-site.xml because
of lack of permission. How can I solve this problem?
> Regards,
> Chiwan Park

View raw message