flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Standalone Cluster vs YARN
Date Wed, 25 Nov 2015 09:11:49 GMT
YARN is not a replacement for Zookeeper. Zookeeper is mandatory to run
Flink in high-availability mode and takes care of leader (JobManager)
election and meta-data persistance.

With YARN, Flink can automatically start new Taskmanagers (and JobManagers)
to compensate for failures. In cluster mode, you need stand-by TMs and JMs
and manually take care that these are "filled-up" again in case of a
failure.

2015-11-25 10:06 GMT+01:00 Welly Tambunan <if05041@gmail.com>:

> Hi Fabian,
>
> Interesting !
>
> However YARN is still tightly couple to HDFS, is that seems wasteful to
> use only YARN without Hadoop ?
>
> Currently we are using Cassandra and CFS ( cass file system )
>
>
> Cheers
>
> On Wed, Nov 25, 2015 at 3:51 PM, Fabian Hueske <fhueske@gmail.com> wrote:
>
>> A strong argument for YARN mode can be the isolation of multiple users
>> and jobs. You can easily start a new Flink cluster for each job or user.
>> However, this comes at the price of resource (memory) fragmentation. YARN
>> mode does not use memory as effective as cluster mode.
>>
>> 2015-11-25 9:46 GMT+01:00 Ufuk Celebi <uce@apache.org>:
>>
>>> > On 25 Nov 2015, at 02:35, Welly Tambunan <if05041@gmail.com> wrote:
>>> >
>>> > Hi All,
>>> >
>>> > I would like to know if there any feature differences between using
>>> Standalone Cluster vs YARN ?
>>> >
>>> > Until now we are using Standalone cluster for our jobs.
>>> > Is there any added value for using YARN ?
>>> >
>>> > We don't have any hadoop infrastructure in place right now but we can
>>> provide that if there's some value to that.
>>>
>>> There are no features, which only work on YARN or in standalone
>>> clusters. YARN mode is essentially starting a standalone cluster in YARN
>>> containers.
>>>
>>> In failure cases I find YARN more convenient, because it takes care of
>>> restarting failed task manager processes/containers for you.
>>>
>>> – Ufuk
>>>
>>>
>>
>
>
> --
> Welly Tambunan
> Triplelands
>
> http://weltam.wordpress.com
> http://www.triplelands.com <http://www.triplelands.com/blog/>
>

Mime
View raw message