flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felipe Cavalcanti <fjfcavalca...@gmail.com>
Subject Re: Should multiple apache flink task managers have strong identity? Also, should I point their state.checkpoints.dir to the same HDFS?
Date Wed, 24 Jan 2018 15:15:31 GMT
Thanks for the answer Patrick!

Awesome! I've made a deployment the following way:

Jobmanager is a statefulset with size 1, I'm using EFS(as a persistent
volume) and zookeeper for persisting it's state across reboots, the
reason it's a statefulset is for having a stable dns for pointing to
it "jobmanager-0.jobmanager"
TaskManagers are deployed as a Deployment with EFS for persisting
checkpoints and snapshots.

It's working great so far.


2018-01-24 13:10 GMT-02:00 Patrick Lucas <patrick@data-artisans.com>:
> Hi Felipe,
> No, using a Deployment for taskmanagers should be fine—they don't need a
> strong identity. For intracluster communication, the taskmanager's hostname
> is used by default, which in most Kubernetes setups is resolvable to the Pod
> IP.
> state.checkpoints.dir should be configured the same for all jobmanagers and
> taskmanagers. All members of the cluster need to be able to see all the
> data, and Flink ensures that things are written to the correct subpath.
> --
> Patrick Lucas
> On Mon, Jan 22, 2018 at 7:23 PM, Felipe Cavalcanti <fjfcavalcanti@gmail.com>
> wrote:
>> Hi,
>> I'm deploying flink to kubernetes and I've some doubts...
>> First one is if the task managers should have strong identity (in
>> which case I will use statefulsets for deploying them). Second one is
>> if I should point rocksdb state.checkpoint.dir in all task managers to
>> the same HDFS path or if each of them should point to their own...
>> Thanks!

View raw message