flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hironori Ogibayashi <ogibaya...@gmail.com>
Subject Re: How to avoid path conflict in zookeeper/HDFS
Date Thu, 04 Aug 2016 07:57:32 GMT
Ufuk,

Thank you for your answer.
I understood only ZooKeeper root path should be different, and I am
glad hear that
YARN will automatically handle the root path in the next release.

Regards,
Hironori

2016-08-04 16:39 GMT+09:00 Ufuk Celebi <uce@apache.org>:
> Hey Hironori,
>
> the storage directories (recovery.zookeeper.storageDir,
> state.backend.fs.checkpointdir) can stay the same I think (either
> random or jobID-specific sub folders should be created there). The
> ZooKeeper root path (recovery.zookeeper.path.root) needs to be unique
> per cluster for HA.
>
> If you upgrade to the to be released 1.1 (vote just passed, binaries
> are being uploaded) this will be set automatically for YARN. You can
> also specify it via the new CLI parameter -z <UNIQUE-NAME> (this sets
> recovery.zookeeper.path.root).
>
> Hope this helps.
>
> Ufuk
>
> On Thu, Aug 4, 2016 at 2:53 AM, Hironori Ogibayashi
> <ogibayashi@gmail.com> wrote:
>> Hello,
>>
>> I have a question about Zookeeper or HDFS paths in case of running
>> Flink on YARN.
>>
>> In my understanding, when I run multiple Flink cluster using the same
>> zookeeper/HDFS, I have to specify different paths, as e.g.
>> state.backend.fs.checkpointdir,
>> recovery.zookeeper.path.root etc.
>>
>> If I run multiple Flink cluster job on YARN, and want to use checkpoint or
>> JobManager HA, do I need to specify different paths for each cluster/job? or
>> does YARN handle this nicely?
>>
>> Regards,
>> Hironori Ogibayashi

Mime
View raw message