Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
From: Dustin Jenkins <djenkins.cadc@gmail.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_3437EC35-3ABC-4A61-9AA2-723E140D2ABF"
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Programmatic configuration with remote JobManager
Message-Id: <0BF5CD2E-5BE8-4C72-BB3F-763E54BA79B8@gmail.com>
Date: Tue, 26 Sep 2017 14:27:09 -0700
To: Flink Users <user@flink.apache.org>
archived-at: Tue, 26 Sep 2017 21:27:25 -0000


--Apple-Mail=_3437EC35-3ABC-4A61-9AA2-723E140D2ABF
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hello,

I=E2=80=99m running a single Flink Job Manager with a Task Manager in =
Docker containers with Java 8.  They are remotely located =
(flink.example.com <http://flink.example.com/>).

I=E2=80=99m submitting a job from my desktop and passing the job to the =
Job Manager with -m flink.example.com:6123 =
<http://flink.example.com:6123/>, which seems to work well.  I=E2=80=99m =
doing a search on an S3 system located at s3.example.com =
<http://s3.example.com/>.

The problem is that in order to have access to the S3 system, the =
$HADOOP_CONFIG/core-site.xml and $FLINK_HOME/flink-conf.yaml need to be =
configured for it at the Job Manager and Task Manager level, which means =
they are tied to that particular endpoint (including my access key and =
secret key).  Is there someway I can specify the configuration only in =
my application so I can leave my Flink server cluster to be mostly =
generic?

Thank you!
Dustin


--Apple-Mail=_3437EC35-3ABC-4A61-9AA2-723E140D2ABF
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">Hello,<div class=3D""><br class=3D""></div><div =
class=3D"">I=E2=80=99m running a single Flink Job Manager with a Task =
Manager in Docker containers with Java 8. &nbsp;They are remotely =
located (<a href=3D"http://flink.example.com" =
class=3D"">flink.example.com</a>).</div><div class=3D""><br =
class=3D""></div><div class=3D"">I=E2=80=99m submitting a job from my =
desktop and passing the job to the Job Manager with -m <a =
href=3D"http://flink.example.com:6123" =
class=3D"">flink.example.com:6123</a>, which seems to work well. =
&nbsp;I=E2=80=99m doing a search on an S3 system located at <a =
href=3D"http://s3.example.com" class=3D"">s3.example.com</a>.</div><div =
class=3D""><br class=3D""></div><div class=3D"">The problem is that in =
order to have access to the S3 system, the $HADOOP_CONFIG/core-site.xml =
and $FLINK_HOME/flink-conf.yaml need to be configured for it at the Job =
Manager and Task Manager level, which means they are tied to that =
particular endpoint (including my access key and secret key). &nbsp;Is =
there someway I can specify the configuration only in my application so =
I can leave my Flink server cluster to be mostly generic?</div><div =
class=3D""><br class=3D""></div><div class=3D"">Thank you!</div><div =
class=3D"">Dustin<br class=3D"">
<br class=3D""></div></body></html>=

--Apple-Mail=_3437EC35-3ABC-4A61-9AA2-723E140D2ABF--