flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From domi...@dbruhn.de
Subject Memory Limits: MiniCluster vs. Local Mode
Date Fri, 03 Mar 2017 16:27:44 GMT
for our CI/CD cycle I'd like to try out our Flink Jobs in an development 
environment without running them against a huge EMR cluster (which is 
what we do for production), so something like a standalone mode.

Until now, for this standalone running, I just started the job jar. As 
the "env.execute()" is in the main-method, this works. I think this is 
callled "Local Mode" by the Flink Devs. I packaged the whole thing in a 
docker container so I have a deployable artefact.

The problem with that is, that the memory constraint seem to be 
difficult to control: Setting Xmx and Xms for the job doesn't seem to 
limit the memory. This is most likely due to flinks off-heap memory 

Now, I got as feedback that perhaps the MiniCluster is the way to go 
instead of the "Local Mode".

My questions:
1. Is the MiniCluster better than the local mode? What are the use-cases 
in which you would choose one over the other?
2. Is there an example how to use the MiniCluster? I see that I need a 
JobGraph, how do I get one?
3. What are the tuning parameters to limit the memory consumption of the 
MiniCluster (and maybe the local mode)?

Thanks for your help,

View raw message