flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Running on a firewalled Yarn cluster?
Date Mon, 02 Nov 2015 15:34:16 GMT
Hi Niels,

so the problem is that you can not submit a job to Flink using the
"/bin/flink" tool, right?
I assume Flink and its TaskManagers properly start and connect to each
other (the number of TaskManagers is shown correctly in the web interface).

I see the following solutions for the problem
a) Add a new page in the job manager web frontend allowing users to upload
and execute a jar with a flink job
b) add options for starting the jobmanager and blob manager on the job
manager container on fixed ports
c) Somehow make the akka rpc requests and blob manager uploads over HTTP
using the YARN proxy

The reason why we use a free port instead a fixed port is that this way two
job manager containers can run on the same machine. So solution b) would
only work if users are not using multiple flink jobs / sessions on yarn at
the same time (or you make somehow sure they are not running on the same
machine).

What's your take on the three solutions?

Does anybody here know how MR is doing it? Are they running the
ApplicationMaster RPC on a fixed port? Do they use HTTP-based calls over
the proxy?

Robert



On Mon, Nov 2, 2015 at 4:05 PM, Niels Basjes <Niels@basjes.nl> wrote:

> Hi,
>
> Here at work our security guys have chosen (long time ago) to only allow
> the firewalls to have the ports open that needed (I say: good call!).
> For the Yarn cluster this includes things like the proxy to see the
> application manager of an application.
> For everything we've done so far (i.e. mr/pig/...) this has worked fine.
>
> Now with Flink I run into problems:
> When I run either the yarn-session or a job on Yarn the application
> manager gets started and I can see the webinterface.
> The problem is that the jobmanager.rpc.address is on one of the worker
> nodes and the jobmanager.rpc.port is essentially a random value.
> A random value which is not accessible because of the firewall rules.
> So I cannot reach the jobmanager on the yarn cluster.
>
> How do I tackle this assuming that opening the all ports on the firewall
> is not an option?
>
> Or is this something that should be handled by Flink? ( Perhaps the
> application manager can proxy the RPC calls? )
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

Mime
View raw message