flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Memory Limits: MiniCluster vs. Local Mode
Date Mon, 06 Mar 2017 14:29:48 GMT
Hi Dominik,

AFAIK, the local mode executions create a mini cluster within the JVM to run the job.

Also, `MiniCluster` seems to be something FLIP-6 related, and since FLIP-6 is still work
in progress, I’m not entirely sure if it is viable at the moment. Right now, you should
look
into using `LocalFlinkMiniCluster`.

In a lot of the Flink integration tests, we use a `LocalFlinkMiniCluster` to setup the test
cluster programatically, and instantiate an `StreamExecutionEnvironment` against that
mini cluster. That would probably be helpful for trying out your Flink jobs programatically
in your CI / CD cycles.

You can also take a look at some Flink test utilities such as
`StreamingMultipleProgramsTestBase`, which helps you to set up an environment
that allows you to submit multiple test jobs on a single `LocalFlinkMiniCluster`. For a
simple example on how to use it, you can take a look at the tests in the Elasticsearch
connector. The Flink Kafka connector tests also have a more complicated test
environment setup where jobs are submitted to the `LocalFlinkMiniCluster` using an
remote environment.

As for memory consumption configuration for the created `LocalFlinkMiniCluster`, I think
you should be able to tune it using the `Configuration` instance passed to it.

Hope this helps!

Cheers,
Gordon

On March 4, 2017 at 12:27:53 AM, dominik@dbruhn.de (dominik@dbruhn.de) wrote:

Hey,  
for our CI/CD cycle I'd like to try out our Flink Jobs in an development  
environment without running them against a huge EMR cluster (which is  
what we do for production), so something like a standalone mode.  

Until now, for this standalone running, I just started the job jar. As  
the "env.execute()" is in the main-method, this works. I think this is  
callled "Local Mode" by the Flink Devs. I packaged the whole thing in a  
docker container so I have a deployable artefact.  

The problem with that is, that the memory constraint seem to be  
difficult to control: Setting Xmx and Xms for the job doesn't seem to  
limit the memory. This is most likely due to flinks off-heap memory  
allocation.  

Now, I got as feedback that perhaps the MiniCluster is the way to go  
instead of the "Local Mode".  

My questions:  
1. Is the MiniCluster better than the local mode? What are the use-cases  
in which you would choose one over the other?  
2. Is there an example how to use the MiniCluster? I see that I need a  
JobGraph, how do I get one?  
3. What are the tuning parameters to limit the memory consumption of the  
MiniCluster (and maybe the local mode)?  

Thanks for your help,  
Dominik  

Mime
View raw message