flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brücke, Christoph <christoph.brue...@campus.tu-berlin.de>
Subject Get TaskManger Memory Configuration from ExecutionEnvironment
Date Tue, 03 May 2016 10:46:21 GMT

I have a problem retrieving `TaskManager` memory configuration settings from the `ExecutionEnvironment`
or I wonder if there is way at all to do so. Basically what I’m trying to do is: I create
FlinkJobs using the Java DataSet API, for that I also need the available memory on the `TaskManager`s,
but *before* creating/running the job.

The options I came up with so far, are:

1. Parse the Flink configuration myself using `GlobalConfiguration#loadConfiguration(String)`.
This has the disadvantage that it doesn’t seem to set default parameters, hence if I use
`LocalFlinkMiniCluster` or some parameters are simply not set in the YAML file they will not
be present in the returned `Configuration` object

2. Create a “dummy” job just to start the `JobManager` and `TaskManager`s and somehow
access the `TaskManagerConfiguration`, here I don’t see an easy way to actually access the
`Taskmanager`s configuration.

3. Use `EnvironmentInformation` to access the memory options from the JVM running the Flink
`TaskManager`. This has the disadvantage that this only gives me a rough estimation on the
actual available memory per `TaskManager`, since I don’t know how many bytes are reserved
for `NetworkBuffer`s etc.

Ultimately I only need the the amount of available memory on the `TaskManger`s that I can
use to store broadcast variables.

Does anyone have an idea how I can retrieve this information ideally before actually running
a Flink job?

Best regards,

View raw message