Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8AE20200BE7 for ; Tue, 6 Dec 2016 03:52:01 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 8970A160B21; Tue, 6 Dec 2016 02:52:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5EF4F160B18 for ; Tue, 6 Dec 2016 03:52:00 +0100 (CET) Received: (qmail 7908 invoked by uid 500); 6 Dec 2016 02:51:54 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 7898 invoked by uid 99); 6 Dec 2016 02:51:54 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Dec 2016 02:51:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E62271881B9 for ; Tue, 6 Dec 2016 02:51:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.648 X-Spam-Level: ** X-Spam-Status: No, score=2.648 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id tgWJB6kjWCxd for ; Tue, 6 Dec 2016 02:51:51 +0000 (UTC) Received: from mail-io0-f180.google.com (mail-io0-f180.google.com [209.85.223.180]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id DA7A75F230 for ; Tue, 6 Dec 2016 02:51:50 +0000 (UTC) Received: by mail-io0-f180.google.com with SMTP id l140so17855628iol.3 for ; Mon, 05 Dec 2016 18:51:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=7mEWirfuKfn7QxhOKIhcX5QAHb1WOAQCDcEJ9jN+tlc=; b=EssIgZTJXDGe9dbpTuwQa3hl6Cohxpe0KyJhX2lB+K384BoklIccANDp0aclk8iv54 7gYSLofBwMt1pqZA6M/EHPx14JLOz4A4+Sjjq6P/igtWrxTS5E6IL0d3mlUYtHQ94GMO opNfHZDJNJQJbpNP6aJROdWavWLOBf+VJ8z7sERjTWCQihOA3bAuiik/XsHeFT0NDqcl R6uKNPBGoMtKYOpBxcrn3n27Q7f61Zc+DR0lBcA/qcRf7Q+8uF3m9msftOcmnlzEdoEQ 3/PxoNahEBKPlZN8DnbUuAlYc+IPqHlLfsnMCDl8txdELRgE6Gb+QLGOHjwL/PdNa7vu /49Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=7mEWirfuKfn7QxhOKIhcX5QAHb1WOAQCDcEJ9jN+tlc=; b=T0yg6scd0zhSXGmj1nIvl8yrzFfk1yRRvMe3QCFl3zYTTVUPWcLWmn0DWaW8LqY/z9 8rCk7zKoHLAQ4zwMeovXn0kMtFU9pCNjmhyCCYvzXXcFGFqpY/6UffK4M0WqLMZelSqr zAdpJxurlNrz713FIVh59NRBqWXQTJKFUJ5jixf+wRWYOkB191O+hgQoorCcROP5YomD dG6VuNLRNiAKjtvco4Kz0s60rW1+qN8/obzapSpJG1S7E2umB8D7L3z7UQdP2vujDrCA fIj24kpLMsgtQNIBNZOOGuvlYfDEsoc0EK8U5b+HtJz+qDHRr4F7s/I3V9dmalobf/BL mW9A== X-Gm-Message-State: AKaTC03xjqk8oKrRCjg/esiSh88oZHzwQfTrcJ1an5X3GVwHVLDTKXETN035DnD5WYeSUSPGPxIO+smRr4ECKQ== X-Received: by 10.107.201.129 with SMTP id z123mr52880147iof.112.1480992707428; Mon, 05 Dec 2016 18:51:47 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Manu Zhang Date: Tue, 06 Dec 2016 02:51:37 +0000 Message-ID: Subject: Re: separation of JVMs for different applications To: user@flink.apache.org Content-Type: multipart/alternative; boundary=94eb2c0b8abc3dc55d0542f47caf archived-at: Tue, 06 Dec 2016 02:52:01 -0000 --94eb2c0b8abc3dc55d0542f47caf Content-Type: text/plain; charset=UTF-8 Thanks Stephan, They don't use YARN now but I think they will consider it. Do you think it would be beneficial to provide such an option as "separate-jvm" in stand-alone mode for streaming processor and long running services ? Or do you think it would introduce too much complexity ? Manu On Tue, Dec 6, 2016 at 1:04 AM Stephan Ewen wrote: > Hi! > > Are your customers using YARN? In that case, the default configuration > will start a new YARN application per Flink job, no JVMs are shared between > jobs. By default, even each slot has its own JVM. > > Greetings, > Stephan > > PS: I think the "spawning new JVMs" is what Till referred to when saying > "spinning up a new cluster". Keep in mind that Flink is also a batch > processor, and it handles sequences of short batch jobs (as issued for > example by interactive shells) and it pre-allocates and manages a lot of > memory for batch jobs. > > > > On Mon, Dec 5, 2016 at 3:48 PM, Manu Zhang > wrote: > > The pro for the multi-tenant cluster mode is that you can share data > between jobs and you don't have to spin up a new cluster for each job. > > > I don't think we have to spin up a new cluster for each job if every job > gets its own JVMs. For examples, Storm will launch a new worker(JVM) for a > new job when free slots are available. How can we share data between jobs > and why ? > > > > On Mon, Dec 5, 2016 at 6:27 PM, Till Rohrmann > wrote: > > The pro for the multi-tenant cluster mode is that you can share data > between jobs and you don't have to spin up a new cluster for each job. This > might be helpful for scenarios where you want to run many short-lived and > light-weight jobs. > > But the important part is that you don't have to use this method. You can > also start a new Flink cluster per job which will then execute the job > isolated from any other jobs (given that you don't submit other jobs to > this cluster). > > Cheers, > Till > > On Sat, Dec 3, 2016 at 2:50 PM, Manu Zhang > wrote: > > Thanks Fabian and Till. > > We have customers who are interested in using Flink but very concerned > about that "multiple jobs share the same set of TMs". I've just joined the > community recently so I'm not sure whether there has been a discussion over > the "multi-tenant cluster mode" before. > > The cons are one job/user's failure may crash another, which is > unacceptable in a multi-tenant scenario. > What are the pros ? Do the pros overweigh the cons ? > > Manu > > On Fri, Dec 2, 2016 at 7:06 PM Till Rohrmann wrote: > > Hi Manu, > > with Flip-6 we will be able to support stricter application isolation by > starting for each job a dedicated JobManager which will execute its tasks > on TM reserved solely for this job. But at the same time we will continue > supporting the multi-tenant cluster mode where tasks belonging to multiple > jobs share the same set of TMs and, thus, might share information between > them. > > Cheers, > Till > > On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske wrote: > > Hi Manu, > > As far as I know, there are not plans to change the stand-alone deployment. > FLIP-6 is focusing on deployments via resource providers (YARN, Mesos, > etc.) which allow to start Flink processes per job. > > Till (in CC) is more familiar with the FLIP-6 effort and might be able to > add more detail. > > Best, > Fabian > > 2016-12-01 4:16 GMT+01:00 Manu Zhang : > > Hi all, > > It seems tasks of different Flink applications can end up in the same JVM > (TaskManager) in standalone mode. Isn't this fragile since errors in one > application could crash another ? I checked FLIP-6 > but > didn't found any mention of changing it in the future. > > Any thoughts or have I missed anything ? > > Thanks, > Manu Zhang > > > > > > > --94eb2c0b8abc3dc55d0542f47caf Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Stephan,

They don't us= e YARN now but I think they will consider it.=C2=A0 Do you think it would b= e beneficial to provide such an option as "separate-jvm" in stand= -alone mode for streaming processor and long running services ? Or do you t= hink it would introduce too much complexity ?

Manu
=

On Tue, Dec 6, 2016 a= t 1:04 AM Stephan Ewen <sewen@apache= .org> wrote:
Hi!

<= /div>
Are your customers using YARN? In that case, = the default configuration will start a new YARN application per Flink job, = no JVMs are shared between jobs. By default, even each slot has its own JVM= .

Greetings,
Stephan

PS: = I think the "spawning new JVMs" is what Till referred to when say= ing "spinning up a new cluster". Keep in mind that Flink is also = a batch processor, and it handles sequences of short batch jobs (as issued = for example by interactive shells) and it pre-allocates and manages a lot o= f memory for batch jobs.



On Mon, Dec 5, 2016 at 3:48 PM, Manu Zhang <owenzhang1990@gmail.com> w= rote:
The pro for the multi-tenant cluster mode is that you = can share data between jobs and you don't have to spin up a new cluster= for each job.

I don't think we have to = spin up a new cluster for each job if every job gets its own JVMs. For exam= ples, Storm will launch a new worker(JVM) for a new job when free slots are= available. How can we share data between jobs and why ?=C2=A0


On Mon, Dec 5, 2016 at 6:27 PM, Till Rohrmann <trohrmann@apache.org> wrote:
The pro for the multi-tenant cluster mode is that you ca= n share data between jobs and you don't have to spin up a new cluster f= or each job. This might be helpful for scenarios where you want to run many= short-lived and light-weight jobs.=C2=A0

But the important part is tha= t you don't have to use this method. You can also start a new Flink clu= ster per job which will then execute the job isolated from any other jobs (= given that you don't submit other jobs to this cluster).

Cheers,
Till

On Sat, = Dec 3, 2016 at 2:50 PM, Manu Zhang &l= t;owenzhang1990@gmail.com> wrote:
Thanks Fabian and Till.

We have customers who are interested in us= ing Flink but very concerned about that "multiple jobs share the same = set of TMs". I've just joined the community recently so I'm no= t sure whether there has been a discussion over the "multi-tenant clus= ter mode" before.=C2=A0

The cons are one job/user's fail= ure may crash another, which is unacceptable in a multi-tenant scenario.
What are the pros ? Do the pros overweigh the c= ons ?

Manu
On Fri, Dec 2, 2016 at 7:06 PM Till Rohrmann <tro= hrmann@apache.org> wrote:
Hi Manu,

with Flip-6 we will be able to support stricter application isolati= on by starting for each job a dedicated JobManager which will execute its t= asks on TM reserved solely for this job. But at the same time we will conti= nue supporting the multi-tenant cluster mode where tasks belonging to multi= ple jobs share the same set of TMs and, thus, might share information betwe= en them.

Cheers,
Till

On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske <fhueske@gm= ail.com> wrote:
Hi M= anu,

As far as I know, there are not plans to change the stand= -alone deployment.
FLIP-6 is focusing on deployments via= resource providers=20 (YARN, Mesos, etc.) which allow to start Flink processes per=20 job.

Till (i= n CC) is more familiar with the FLIP-6 effort and might be able to add more= detail.

Best,
Fabian
2016-12-01 4:16 GMT+01:00 Manu Zhang <ow= enzhang1990@gmail.com>:
Hi all,
It seems tasks of diffe= rent Flink applications can end up in the same JVM (TaskManager) in standal= one mode. Isn't this fragile since errors in one application could cras= h another ? I checked FLIP-6=C2=A0but didn't found any mention of chang= ing it in the future.=C2=A0
Any thoughts or have I missed anything ?

Thanks,
Manu Zhang


=



--94eb2c0b8abc3dc55d0542f47caf--