flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: separation of JVMs for different applications
Date Fri, 09 Dec 2016 11:16:53 GMT
Hi Manu,

afaik there is no JIRA for standalone v2.0 yet. So feel free to open an
JIRA for it.

Just a small correction, FLIP-6 is not almost finished yet. But we're
working on it and are happy for every helping hand :-)

Cheers,
Till

On Fri, Dec 9, 2016 at 2:27 AM, Manu Zhang <owenzhang1990@gmail.com> wrote:

> If there are not any existing jira for standalone v2.0, may I open a new
> one ?
>
> Thanks,
> Manu
>
> On Wed, Dec 7, 2016 at 12:39 PM Manu Zhang <owenzhang1990@gmail.com>
> wrote:
>
>> Good to know that.
>>
>> Is it the "standalone setup v2.0" section ? The wiki page has no
>> Google-Doc-like change histories.
>> Any jiras opened for that ? Not sure that will be noticed given FLIP-6 is
>> almost finished.
>>
>> Thanks,
>> Manu
>>
>> On Tue, Dec 6, 2016 at 11:55 PM Stephan Ewen <sewen@apache.org> wrote:
>>
>> Hi!
>>
>> We are currently changing the resource and process model quite a bit:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077
>> As part of that, I think it makes sense to introduce something like that.
>>
>> What you can do today is to set TaskManagers to use one slot only, and
>> then start multiple TaskManagers per machine. That makes sure that JVMs are
>> never shared across machines.
>> If you use the "start-cluster.sh" script from Flink, you can enter the
>> same hostname multiple times in the workers file, and it will start
>> multiple TaskManagers on a machine.
>>
>> Best,
>> Stephan
>>
>>
>>
>> On Tue, Dec 6, 2016 at 3:51 AM, Manu Zhang <owenzhang1990@gmail.com>
>> wrote:
>>
>> Thanks Stephan,
>>
>> They don't use YARN now but I think they will consider it.  Do you think
>> it would be beneficial to provide such an option as "separate-jvm" in
>> stand-alone mode for streaming processor and long running services ? Or do
>> you think it would introduce too much complexity ?
>>
>> Manu
>>
>> On Tue, Dec 6, 2016 at 1:04 AM Stephan Ewen <sewen@apache.org> wrote:
>>
>> Hi!
>>
>> Are your customers using YARN? In that case, the default configuration
>> will start a new YARN application per Flink job, no JVMs are shared between
>> jobs. By default, even each slot has its own JVM.
>>
>> Greetings,
>> Stephan
>>
>> PS: I think the "spawning new JVMs" is what Till referred to when saying
>> "spinning up a new cluster". Keep in mind that Flink is also a batch
>> processor, and it handles sequences of short batch jobs (as issued for
>> example by interactive shells) and it pre-allocates and manages a lot of
>> memory for batch jobs.
>>
>>
>>
>> On Mon, Dec 5, 2016 at 3:48 PM, Manu Zhang <owenzhang1990@gmail.com>
>> wrote:
>>
>> The pro for the multi-tenant cluster mode is that you can share data
>> between jobs and you don't have to spin up a new cluster for each job.
>>
>>
>> I don't think we have to spin up a new cluster for each job if every job
>> gets its own JVMs. For examples, Storm will launch a new worker(JVM) for a
>> new job when free slots are available. How can we share data between jobs
>> and why ?
>>
>>
>>
>> On Mon, Dec 5, 2016 at 6:27 PM, Till Rohrmann <trohrmann@apache.org>
>> wrote:
>>
>> The pro for the multi-tenant cluster mode is that you can share data
>> between jobs and you don't have to spin up a new cluster for each job. This
>> might be helpful for scenarios where you want to run many short-lived and
>> light-weight jobs.
>>
>> But the important part is that you don't have to use this method. You can
>> also start a new Flink cluster per job which will then execute the job
>> isolated from any other jobs (given that you don't submit other jobs to
>> this cluster).
>>
>> Cheers,
>> Till
>>
>> On Sat, Dec 3, 2016 at 2:50 PM, Manu Zhang <owenzhang1990@gmail.com>
>> wrote:
>>
>> Thanks Fabian and Till.
>>
>> We have customers who are interested in using Flink but very concerned
>> about that "multiple jobs share the same set of TMs". I've just joined the
>> community recently so I'm not sure whether there has been a discussion over
>> the "multi-tenant cluster mode" before.
>>
>> The cons are one job/user's failure may crash another, which is
>> unacceptable in a multi-tenant scenario.
>> What are the pros ? Do the pros overweigh the cons ?
>>
>> Manu
>>
>> On Fri, Dec 2, 2016 at 7:06 PM Till Rohrmann <trohrmann@apache.org>
>> wrote:
>>
>> Hi Manu,
>>
>> with Flip-6 we will be able to support stricter application isolation by
>> starting for each job a dedicated JobManager which will execute its tasks
>> on TM reserved solely for this job. But at the same time we will continue
>> supporting the multi-tenant cluster mode where tasks belonging to multiple
>> jobs share the same set of TMs and, thus, might share information between
>> them.
>>
>> Cheers,
>> Till
>>
>> On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske <fhueske@gmail.com> wrote:
>>
>> Hi Manu,
>>
>> As far as I know, there are not plans to change the stand-alone
>> deployment.
>> FLIP-6 is focusing on deployments via resource providers (YARN, Mesos,
>> etc.) which allow to start Flink processes per job.
>>
>> Till (in CC) is more familiar with the FLIP-6 effort and might be able to
>> add more detail.
>>
>> Best,
>> Fabian
>>
>> 2016-12-01 4:16 GMT+01:00 Manu Zhang <owenzhang1990@gmail.com>:
>>
>> Hi all,
>>
>> It seems tasks of different Flink applications can end up in the same JVM
>> (TaskManager) in standalone mode. Isn't this fragile since errors in one
>> application could crash another ? I checked FLIP-6
>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077>
but
>> didn't found any mention of changing it in the future.
>>
>> Any thoughts or have I missed anything ?
>>
>> Thanks,
>> Manu Zhang
>>
>>
>>
>>
>>
>>
>>
>>

Mime
View raw message