flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Lucas <patr...@data-artisans.com>
Subject Re: Should multiple apache flink task managers have strong identity? Also, should I point their state.checkpoints.dir to the same HDFS?
Date Wed, 24 Jan 2018 15:10:47 GMT
Hi Felipe,

No, using a Deployment for taskmanagers should be fine—they don't need a
strong identity. For intracluster communication, the taskmanager's hostname
is used by default, which in most Kubernetes setups is resolvable to the
Pod IP.

state.checkpoints.dir should be configured the same for all jobmanagers and
taskmanagers. All members of the cluster need to be able to see all the
data, and Flink ensures that things are written to the correct subpath.

Patrick Lucas

On Mon, Jan 22, 2018 at 7:23 PM, Felipe Cavalcanti <fjfcavalcanti@gmail.com>

> Hi,
> I'm deploying flink to kubernetes and I've some doubts...
> First one is if the task managers should have strong identity (in
> which case I will use statefulsets for deploying them). Second one is
> if I should point rocksdb state.checkpoint.dir in all task managers to
> the same HDFS path or if each of them should point to their own...
> Thanks!

View raw message