flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <owenzhang1...@gmail.com>
Subject Re: separation of JVMs for different applications
Date Mon, 05 Dec 2016 14:48:24 GMT
>
> The pro for the multi-tenant cluster mode is that you can share data
> between jobs and you don't have to spin up a new cluster for each job.


I don't think we have to spin up a new cluster for each job if every job
gets its own JVMs. For examples, Storm will launch a new worker(JVM) for a
new job when free slots are available. How can we share data between jobs
and why ?



On Mon, Dec 5, 2016 at 6:27 PM, Till Rohrmann <trohrmann@apache.org> wrote:

> The pro for the multi-tenant cluster mode is that you can share data
> between jobs and you don't have to spin up a new cluster for each job. This
> might be helpful for scenarios where you want to run many short-lived and
> light-weight jobs.
>
> But the important part is that you don't have to use this method. You can
> also start a new Flink cluster per job which will then execute the job
> isolated from any other jobs (given that you don't submit other jobs to
> this cluster).
>
> Cheers,
> Till
>
> On Sat, Dec 3, 2016 at 2:50 PM, Manu Zhang <owenzhang1990@gmail.com>
> wrote:
>
>> Thanks Fabian and Till.
>>
>> We have customers who are interested in using Flink but very concerned
>> about that "multiple jobs share the same set of TMs". I've just joined the
>> community recently so I'm not sure whether there has been a discussion over
>> the "multi-tenant cluster mode" before.
>>
>> The cons are one job/user's failure may crash another, which is
>> unacceptable in a multi-tenant scenario.
>> What are the pros ? Do the pros overweigh the cons ?
>>
>> Manu
>>
>> On Fri, Dec 2, 2016 at 7:06 PM Till Rohrmann <trohrmann@apache.org>
>> wrote:
>>
>>> Hi Manu,
>>>
>>> with Flip-6 we will be able to support stricter application isolation by
>>> starting for each job a dedicated JobManager which will execute its tasks
>>> on TM reserved solely for this job. But at the same time we will continue
>>> supporting the multi-tenant cluster mode where tasks belonging to multiple
>>> jobs share the same set of TMs and, thus, might share information between
>>> them.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske <fhueske@gmail.com>
>>> wrote:
>>>
>>> Hi Manu,
>>>
>>> As far as I know, there are not plans to change the stand-alone
>>> deployment.
>>> FLIP-6 is focusing on deployments via resource providers (YARN, Mesos,
>>> etc.) which allow to start Flink processes per job.
>>>
>>> Till (in CC) is more familiar with the FLIP-6 effort and might be able
>>> to add more detail.
>>>
>>> Best,
>>> Fabian
>>>
>>> 2016-12-01 4:16 GMT+01:00 Manu Zhang <owenzhang1990@gmail.com>:
>>>
>>> Hi all,
>>>
>>> It seems tasks of different Flink applications can end up in the same
>>> JVM (TaskManager) in standalone mode. Isn't this fragile since errors in
>>> one application could crash another ? I checked FLIP-6
>>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077>
but
>>> didn't found any mention of changing it in the future.
>>>
>>> Any thoughts or have I missed anything ?
>>>
>>> Thanks,
>>> Manu Zhang
>>>
>>>
>>>
>>>
>

Mime
View raw message