From user-return-23132-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Sep 24 11:25:16 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id CE40E180649 for ; Mon, 24 Sep 2018 11:25:15 +0200 (CEST) Received: (qmail 48623 invoked by uid 500); 24 Sep 2018 09:25:14 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 48613 invoked by uid 99); 24 Sep 2018 09:25:14 -0000 Received: from mail-relay.apache.org (HELO mailrelay2-lw-us.apache.org) (207.244.88.137) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Sep 2018 09:25:14 +0000 Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by mailrelay2-lw-us.apache.org (ASF Mail Server at mailrelay2-lw-us.apache.org) with ESMTPSA id A4D6D22FA for ; Mon, 24 Sep 2018 09:25:13 +0000 (UTC) Received: by mail-qk1-f176.google.com with SMTP id a189-v6so1474914qkb.2 for ; Mon, 24 Sep 2018 02:25:13 -0700 (PDT) X-Gm-Message-State: ABuFfoi5kzt7QjDwp9Uc9xhZ3R0MYJ68S9uioTxlomComLAOPzK6PhOr w/b9j2HTnvpjZX+UCKNVeg/9qWQMZGLibIMbfhA= X-Google-Smtp-Source: ACcGV61mw+becqjOhTXl5KySlW23wzNMD5R/47hufv8hkl+rIFQPkvvjpNQ1j8pKKAO20W0iqBSr+r/brA0XkZ7l+eA= X-Received: by 2002:a37:6396:: with SMTP id x144-v6mr3974844qkb.340.1537781113310; Mon, 24 Sep 2018 02:25:13 -0700 (PDT) MIME-Version: 1.0 References: <37AFF4942523AA40A65B01F1A7B4B82A0F6F896F@GBWGCEUHUBD0103.rbsres07.net> <9cfe79d1-15d2-7db3-fa09-0853ecd54190@apache.org> In-Reply-To: From: Till Rohrmann Date: Mon, 24 Sep 2018 11:24:36 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Running Flink in Google Cloud Platform (GCP) - can Flink be truly elastic? To: konstantin@data-artisans.com Cc: Alexander.Gardner@rbs.com, user , Gary Yao , dwysakowicz@apache.org Content-Type: multipart/alternative; boundary="00000000000000557905769a9111" --00000000000000557905769a9111 Content-Type: text/plain; charset="UTF-8" Hi Alexander, the issue for the reactive mode, the mode which reacts to newly available resources and scales the up accordingly, is here: https://issues.apache.org/jira/browse/FLINK-10407. It does not contain a lot of details but we are actively working on publishing the corresponding design document soon. See also https://issues.apache.org/jira/browse/FLINK-10404 which is related to the reactive mode. Cheers, Till On Sun, Sep 23, 2018 at 5:33 PM Konstantin Knauf < konstantin@data-artisans.com> wrote: > Hi Alexander, > > broadly speaking, what you are doing right now, is in line with what is > currently possible with Apache Flink. Can you share a little bit more > information about your setup (K8s/Flink-Standalone? > Job-Mode/Session-Mode?)? You might find Gary's Flink Forward [1] talk > interesting. He demonstrates how a Flink job automatically scales out, when > it is given more resources by the resource manager, e.g. Kubernetes. But > this is still work-in-progress. > > Best, > > Konstantin > > [1] > https://data-artisans.com/flink-forward-berlin/resources/flink-as-a-library-and-still-as-a-framework > > > On Fri, Sep 21, 2018 at 5:42 PM Dawid Wysakowicz > wrote: > >> Hi Alexander, >> >> I've redirected your question to user mailing list. The goal of >> community list is for "Broader community discussions related to meetups, >> conferences, blog posts and job offers" >> >> Quick answer to your question is that dynamic scaling of flink job's is >> a work in progress. Maybe Gary or Till cc'ed can share some more details >> on that topic. >> >> Best, >> >> Dawid >> >> >> On 21/09/18 17:25, Alexander.Gardner@rbs.com.INVALID wrote: >> > Hi >> > >> > I'm trying to understand what it means to run a Flink cluster inside >> the Google Cloud Platform and whether it can act in an "elastic" way; if >> the cluster needs more resources to accommodate a sudden demand or increase >> in Flink jobs, will GCP automatically detect this and spool up more Task >> Managers to provide extra task slots? >> > >> > If we consider the following two simple use cases, how would GCP >> address them? >> > >> > >> > 1) No free task slots to run new flink jobs >> > >> > 2) A slow flink job needs an increased parallelism to improve >> throughput >> > >> > Currently, we'd handle the above use cases by: >> > >> > >> > 1) knowing that the job failed due to "no free slots", check the >> exception text, schedule to add a new task manager and rerun the job, >> knowing that there are now available task slots. >> > >> > 2) We'd monitor the speed of the job ourselves, stop the job, >> specify which components (operators) in the stream reqd an increase in >> parallelism (for example via job properties), then relaunch the job; if not >> enough slots were available, we'd have to consider adding extra task >> managers. >> > >> > >> > So my question is...can Google Cloud Platform (GCP) automatically >> launch extra TMs to handle the above? >> > >> > If we proposed to run a Flink cluster in a GCP container, can GCP make >> Flink behave dynamically elastic in the same way that Google DataFlow >> apparently can? >> > >> > Regards >> > >> > >> > Alex >> > >> > >> > The Royal Bank of Scotland plc. Registered in Scotland No 83026. >> Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB. The Royal Bank >> of Scotland is authorised by the Prudential Regulation Authority, and >> regulated by the Financial Conduct Authority and Prudential Regulation >> Authority. The Royal Bank of Scotland N.V. is authorised and regulated by >> the De Nederlandsche Bank and has its seat at Amsterdam, the Netherlands, >> and is registered in the Commercial Register under number 33002587. >> Registered Office: Gustav Mahlerlaan 350, Amsterdam, The Netherlands. The >> Royal Bank of Scotland N.V. and The Royal Bank of Scotland plc are >> authorised to act as agent for each other in certain jurisdictions. >> > >> > National Westminster Bank Plc. Registered in England No. 929027. >> Registered Office: 135 Bishopsgate, London EC2M 3UR. National Westminster >> Bank Plc is authorised by the Prudential Regulation Authority, and >> regulated by the Financial Conduct Authority and the Prudential Regulation >> Authority. >> > >> > The Royal Bank of Scotland plc and National Westminster Bank Plc are >> authorised to act as agent for each other. >> > >> > This e-mail message is confidential and for use by the addressee only. >> If the message is received by anyone other than the addressee, please >> return the message to the sender by replying to it and then delete the >> message from your computer. Internet e-mails are not necessarily secure. >> The Royal Bank of Scotland plc, The Royal Bank of Scotland N.V., National >> Westminster Bank Plc or any affiliated entity (RBS or us) does not accept >> responsibility for changes made to this message after it was sent. RBS may >> monitor e-mails for business and operational purposes. By replying to this >> message you understand that the content of your message may be monitored. >> > >> > Whilst all reasonable care has been taken to avoid the transmission of >> viruses, it is the responsibility of the recipient to ensure that the >> onward transmission, opening or use of this message and any attachments >> will not adversely affect its systems or data. No responsibility is >> accepted by RBS in this regard and the recipient should carry out such >> virus and other checks as it considers appropriate. >> > >> > Visit our website at www.rbs.com >> > >> >> >> > > -- > > Konstantin Knauf | Solution Architect > > data Artisans > > > Follow us @dataArtisans > > -- > > Join Flink Forward - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Data Artisans GmbH | Stresemannstr. 121A,10963 Berlin, Germany > > data Artisans, Inc. | 1161 Mission Street, San Francisco, CA-94103, USA > > > -- > Data Artisans GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen > --00000000000000557905769a9111 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Alexander,

the issue for the reactive mode, the mode which reacts to newly av= ailable resources and scales the up accordingly, is here:=C2=A0https://issues.apache.org= /jira/browse/FLINK-10407. It does not contain a lot of details but we a= re actively working on publishing the corresponding design document soon. S= ee also https= ://issues.apache.org/jira/browse/FLINK-10404 which is related to the re= active mode.

Cheers,
Till

On Sun, Sep 23, 20= 18 at 5:33 PM Konstantin Knauf <konstantin@data-artisans.com> wrote:
Hi Alexander,
<= /div>

broadly speaking, what you are doing right now, is= in line with what is currently possible with Apache Flink. Can you share a= little bit more information about your setup (K8s/Flink-Standalone? Job-Mo= de/Session-Mode?)? You might find Gary's Flink Forward [1] talk interes= ting. He demonstrates how a Flink job automatically scales out, when it is = given more resources by the resource manager, e.g. Kubernetes. But this is = still work-in-progress.

Best,
=
Konstantin

[1] https://data-artisans.com/flink-for= ward-berlin/resources/flink-as-a-library-and-still-as-a-framework


= On Fri, Sep 21, 2018 at 5:42 PM Dawid Wysakowicz <dwysakowicz@apache.org> wrote:=
Hi Alexander,

I've redirected your question to user mailing list. The goal of
community list is for "Broader community discussions related to meetup= s,
conferences, blog posts and job offers"

Quick answer to your question is that dynamic scaling of flink job's is=
a work in progress. Maybe Gary or Till cc'ed can share some more detail= s
on that topic.

Best,

Dawid


On 21/09/18 17:25, Alexander.Gardner@rbs.com.INVALID wrote:
> Hi
>
> I'm trying to understand what it means to run a Flink cluster insi= de the Google Cloud Platform and whether it can act in an "elastic&quo= t; way; if the cluster needs more resources to accommodate a sudden demand = or increase in Flink jobs, will GCP automatically detect this and spool up = more Task Managers to provide extra task slots?
>
> If we consider the following two simple use cases, how would GCP addre= ss them?
>
>
> 1)=C2=A0 =C2=A0 =C2=A0No free task slots to run new flink jobs
>
> 2)=C2=A0 =C2=A0 =C2=A0A slow flink job needs an increased parallelism = to improve throughput
>
> Currently, we'd handle the above use cases by:
>
>
> 1)=C2=A0 =C2=A0 =C2=A0knowing that the job failed due to "no free= slots", check the exception text, schedule to add a new task manager = and rerun the job, knowing that there are now available task slots.
>
> 2)=C2=A0 =C2=A0 =C2=A0We'd monitor the speed of the job ourselves,= stop the job, specify which components (operators) in the stream reqd an i= ncrease in parallelism (for example via job properties), then relaunch the = job; if not enough slots were available, we'd have to consider adding e= xtra task managers.
>
>
> So my question is...can Google Cloud Platform (GCP) automatically laun= ch extra TMs to handle the above?
>
> If we proposed to run a Flink cluster in a GCP container, can GCP make= Flink behave dynamically elastic in the same way that Google DataFlow appa= rently can?
>
> Regards
>
>
> Alex
>
>
> The Royal Bank of Scotland plc. Registered in Scotland No 83026. Regis= tered Office: 36 St Andrew Square, Edinburgh EH2 2YB. The Royal Bank of Sco= tland is authorised by the Prudential Regulation Authority, and regulated b= y the Financial Conduct Authority and Prudential Regulation Authority. The = Royal Bank of Scotland N.V. is authorised and regulated by the De Nederland= sche Bank and has its seat at Amsterdam, the Netherlands, and is registered= in the Commercial Register under number 33002587. Registered Office: Gusta= v Mahlerlaan 350, Amsterdam, The Netherlands. The Royal Bank of Scotland N.= V. and The Royal Bank of Scotland plc are authorised to act as agent for ea= ch other in certain jurisdictions.
>
> National Westminster Bank Plc.=C2=A0 Registered in England No. 929027.= =C2=A0 Registered Office: 135 Bishopsgate, London EC2M 3UR.=C2=A0 National = Westminster Bank Plc is authorised by the Prudential Regulation Authority, = and regulated by the Financial Conduct Authority and the Prudential Regulat= ion Authority.
>
> The Royal Bank of Scotland plc and National Westminster Bank Plc are a= uthorised to act as agent for each other.
>
> This e-mail message is confidential and for use by the addressee only.= =C2=A0 If the message is received by anyone other than the addressee, pleas= e return the message to the sender by replying to it and then delete the me= ssage from your computer.=C2=A0 Internet e-mails are not necessarily secure= .=C2=A0 The Royal Bank of Scotland plc, The Royal Bank of Scotland N.V., Na= tional Westminster Bank Plc or any affiliated entity (RBS or us) does not a= ccept responsibility for changes made to this message after it was sent.=C2= =A0 RBS may monitor e-mails for business and operational purposes.=C2=A0 By= replying to this message you understand that the content of your message m= ay be monitored.
>
> Whilst all reasonable care has been taken to avoid the transmission of= viruses, it is the responsibility of the recipient to ensure that the onwa= rd transmission, opening or use of this message and any attachments will no= t adversely affect its systems or data.=C2=A0 No responsibility is accepted= by RBS in this regard and the recipient should carry out such virus and ot= her checks as it considers appropriate.
>
> Visit our website at www.rbs.com <http://www.rbs.com/>
>




--

Konstantin Knauf | Solution Architect

data Artisans
<= /a>

Follow us @dataArtisans

--

Join Flink = Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Data= Artisans GmbH | Stresemannstr. 121A,10963 Berlin, Germany
data Artisans, Inc. | = 1161 Mis= sion Street, San Francisco, CA-94103, USA

--

Data Artisans Gmb= H
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managi= ng Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
--00000000000000557905769a9111--