flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Baghino <stefano.bagh...@radicalbit.io>
Subject Re: HDFS namenode and Flink
Date Mon, 23 May 2016 10:57:58 GMT
One last quick note: if you're going to run individual jobs on YARN instead
of a long running session, make sure you provide each job with a separate
set of directories for (surely) ZK storage and (possibly*) state backend,
otherwise the state of the jobs will end up entangled and you may
experience some undefined behavior.

* I'm not really sure about this last one, perhaps some more experienced ML
user can help me out on this.

On Mon, May 23, 2016 at 12:54 PM, Stefano Baghino <
stefano.baghino@radicalbit.io> wrote:

> I think the only keys of interest for your needs (highly available with
> HDFS state backend) are
>
> state.backend: filesystem
> state.backend.fs.checkpointdir: hdfs:///path/to/checkpoints # fill in
> according to your needs
> recovery.zookeeper.storageDir: /path/to/znode # again, fill in according
> to your needs
> recovery.mode: zookeeper
> recovery.zookeeper.quorum:
> zk-ensemble-1:2181,zk-ensemble-2:2181,zk-ensemble-3:2181 # put your zk
> ensemble here
>
> If these keys are set you should be good to go. I hope I've been of some
> help. :)
>
> On Mon, May 23, 2016 at 12:37 PM, <thomas@lamirault.net> wrote:
>
>> Hello flinkers,
>>
>> We will activate namenode HDFS high availability in our cluster, and I
>> want to know if there is additional configuration for flink ?
>> We actually use YARN for launching our flink application, and hdfs
>> filesystem to store the state backend
>>
>> Thanks
>>
>> Thomas
>>
>
>
>
> --
> BR,
> Stefano Baghino
>
> Software Engineer @ Radicalbit
>



-- 
BR,
Stefano Baghino

Software Engineer @ Radicalbit

Mime
View raw message