flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bostow, Ben" <ben.bos...@here.com>
Subject Programmatically Creating a Flink Cluster On YARN
Date Tue, 02 Aug 2016 21:09:17 GMT
I’m currently trying to programmatically create a Flink cluster on a given YARN cluster.
I’m using the FlinkYarnClientBase class to do this currently with some limitations (Flink
version 1.0.3).

I’m wanting to pass in my own YARN configuration so that I can deploy Flink on different
YARN clusters. Currently these appear to be picked up from an environment variable YARN_CONF
or if I add the yarn configuration files into the classpath it works for that one.

Is there a way I can dynamically build the YARN configuration and Flink configuration without
utilizing the file system? It would be nice if we had the option to utilize the same Flink
client code and have the CLI be a wrapper that adds all the environment variable stuff, and
then when we need to do create these programmatically we can do that too.

I’ve started looking at the new code in Flink master and it seems a little easier than in
1.0.3 but seems that the YarnConfiguration is private in AbstractYarnClusterDescriptor making
it difficult to set the configuration programmatically. I’m not sure if the new client code
still throws an exception if the Flink config doesn’t exist, even though I could build out
the Flink config programmatically.

Is there a recommended approach to allow this?

Benjamin
Mime
View raw message