flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: separation of JVMs for different applications
Date Mon, 05 Dec 2016 10:27:50 GMT
The pro for the multi-tenant cluster mode is that you can share data
between jobs and you don't have to spin up a new cluster for each job. This
might be helpful for scenarios where you want to run many short-lived and
light-weight jobs.

But the important part is that you don't have to use this method. You can
also start a new Flink cluster per job which will then execute the job
isolated from any other jobs (given that you don't submit other jobs to
this cluster).

Cheers,
Till

On Sat, Dec 3, 2016 at 2:50 PM, Manu Zhang <owenzhang1990@gmail.com> wrote:

> Thanks Fabian and Till.
>
> We have customers who are interested in using Flink but very concerned
> about that "multiple jobs share the same set of TMs". I've just joined the
> community recently so I'm not sure whether there has been a discussion over
> the "multi-tenant cluster mode" before.
>
> The cons are one job/user's failure may crash another, which is
> unacceptable in a multi-tenant scenario.
> What are the pros ? Do the pros overweigh the cons ?
>
> Manu
>
> On Fri, Dec 2, 2016 at 7:06 PM Till Rohrmann <trohrmann@apache.org> wrote:
>
>> Hi Manu,
>>
>> with Flip-6 we will be able to support stricter application isolation by
>> starting for each job a dedicated JobManager which will execute its tasks
>> on TM reserved solely for this job. But at the same time we will continue
>> supporting the multi-tenant cluster mode where tasks belonging to multiple
>> jobs share the same set of TMs and, thus, might share information between
>> them.
>>
>> Cheers,
>> Till
>>
>> On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske <fhueske@gmail.com> wrote:
>>
>> Hi Manu,
>>
>> As far as I know, there are not plans to change the stand-alone
>> deployment.
>> FLIP-6 is focusing on deployments via resource providers (YARN, Mesos,
>> etc.) which allow to start Flink processes per job.
>>
>> Till (in CC) is more familiar with the FLIP-6 effort and might be able to
>> add more detail.
>>
>> Best,
>> Fabian
>>
>> 2016-12-01 4:16 GMT+01:00 Manu Zhang <owenzhang1990@gmail.com>:
>>
>> Hi all,
>>
>> It seems tasks of different Flink applications can end up in the same JVM
>> (TaskManager) in standalone mode. Isn't this fragile since errors in one
>> application could crash another ? I checked FLIP-6
>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077>
but
>> didn't found any mention of changing it in the future.
>>
>> Any thoughts or have I missed anything ?
>>
>> Thanks,
>> Manu Zhang
>>
>>
>>
>>

Mime
View raw message