flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dustin Jenkins <djenkins.c...@gmail.com>
Subject Programmatic configuration with remote JobManager
Date Tue, 26 Sep 2017 21:27:09 GMT

I’m running a single Flink Job Manager with a Task Manager in Docker containers with Java
8.  They are remotely located (flink.example.com <http://flink.example.com/>).

I’m submitting a job from my desktop and passing the job to the Job Manager with -m flink.example.com:6123
<http://flink.example.com:6123/>, which seems to work well.  I’m doing a search on
an S3 system located at s3.example.com <http://s3.example.com/>.

The problem is that in order to have access to the S3 system, the $HADOOP_CONFIG/core-site.xml
and $FLINK_HOME/flink-conf.yaml need to be configured for it at the Job Manager and Task Manager
level, which means they are tied to that particular endpoint (including my access key and
secret key).  Is there someway I can specify the configuration only in my application so I
can leave my Flink server cluster to be mostly generic?

Thank you!

View raw message