Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A1AF4200B64 for ; Tue, 2 Aug 2016 16:26:37 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A02E7160A76; Tue, 2 Aug 2016 14:26:37 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6FBE6160A65 for ; Tue, 2 Aug 2016 16:26:36 +0200 (CEST) Received: (qmail 425 invoked by uid 500); 2 Aug 2016 14:26:35 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 416 invoked by uid 99); 2 Aug 2016 14:26:35 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2016 14:26:35 +0000 Received: from mail-qk0-f178.google.com (mail-qk0-f178.google.com [209.85.220.178]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id F1D721A01A7 for ; Tue, 2 Aug 2016 14:26:33 +0000 (UTC) Received: by mail-qk0-f178.google.com with SMTP id v123so43265489qkh.3 for ; Tue, 02 Aug 2016 07:26:33 -0700 (PDT) X-Gm-Message-State: AEkoouvHqgYsMknZAdRvujEtylWXJdm+Jv1bIhkYQWy51PU+YUhFwN0RnCYR5EWPU4i386t4PySlFMFSdHB1KA== X-Received: by 10.55.176.130 with SMTP id z124mr78527736qke.164.1470147993072; Tue, 02 Aug 2016 07:26:33 -0700 (PDT) MIME-Version: 1.0 Received: by 10.55.10.136 with HTTP; Tue, 2 Aug 2016 07:26:32 -0700 (PDT) In-Reply-To: <9633659C9F9D924DB69F4F53CDF42743BD1D2F@kasasiserver> References: <9633659C9F9D924DB69F4F53CDF42743BD1CFE@kasasiserver> <9633659C9F9D924DB69F4F53CDF42743BD1D2F@kasasiserver> From: Till Rohrmann Date: Tue, 2 Aug 2016 22:26:32 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: partial savepoints/combining savepoints To: user@flink.apache.org Content-Type: multipart/alternative; boundary=94eb2c06fb2ee502800539178002 archived-at: Tue, 02 Aug 2016 14:26:37 -0000 --94eb2c06fb2ee502800539178002 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Claudia, 1) At the moment the offset information will be written to the ZooKeeper quorum used by Kafka as well as to the savepoint. Reading the savepoint is not so easy to do since you would need to know the internal representation of the savepoint. But you could try to read the Kafka offsets from ZooKeeper. 2) That depends a little bit on the deployment and the size of the job. Are you using a yarn session or a standalone cluster? Then the task manager should already be registered at the job manager and the deployment for each task should be in the milli second range. If you start a new yarn application for a flink job (per job cluster), then it might take a bit longer depending on how long it takes to allocate the requested resources by Yarn. But once this is done, the deployment for a task should be in the sub second range. 3) If you want to keep the different Flink jobs separated, then you should submit them separately to a Flink cluster or start a Flink cluster per job (e.g. with Yarn). I don't think that this a bad architecture if you want to fulfil these requirements. However, I'm not sure whether merging and splitting savepoints will be implemented anytime soon. Actually we're currently working on improving Flink's functionality to be started with a dedicated job. This means that you start a job manager which has already the job jar in its classpath and directly starts executing the contained job. This will be helpful for deployment scenarios how they appear when using docker images, for example. I could imagine that this could be helpful for your use case as well. Cheers, Till On Mon, Aug 1, 2016 at 10:40 PM, Claudia Wegmann wrote: > Hi Till, > > > > thanks for the quick reply. Too bad, I thought I was on the right track > with savepoints here. > > > > Some follow-up questions: > > > > 1.) Can I do the whole thing of transferring state and the position in > the Kafka topic manually for one stream? In other words: is this > information accessible easily? > > 2.) In any case I would need to stop the running job, change the > topology (e.g. the number of streams in the program) and resume processin= g. > Can you name the overhead of time coming from stopping and starting a Fli= nk > job? > > 3.) I=E2=80=99m aware of the upcoming feature for scaling in and out. = But I > don=E2=80=99t quite see, how this will help me with different services. > I thought of each service having its own Flink instance/cluster. I would > commit this service as one job to the dedicated Flink containing all the > necessary streams and computations. Is this a bad architecture? > Would it be better to have one big Flink cluster and commit one big Job, > which contains all the streams? (As I got to know, committing multiple jo= bs > to one Flink instance is not recommended). > To be honest, I=E2=80=99m not quite there to totally understand the diffe= rent > deployment options of Flink and how to bring them together with a > microservice architecture where I have a service packed as a JAR-File and > wanting to be able to just deploy this JAR-File. I thought of this servic= e > containing Flink and then start the JobManager and some TaskManagers from > this service and deploy itself as the Flink job with a dedicated entry > point. Is this a good idea? Or is it even possible? > > > > Thanks in advance, > > Claudia > > > > *Von:* Till Rohrmann [mailto:trohrmann@apache.org] > *Gesendet:* Montag, 1. August 2016 16:21 > *An:* user@flink.apache.org > *Betreff:* Re: partial savepoints/combining savepoints > > > > Hi Claudia, > > > > unfortunately neither taking partial savepoints nor combining multiple > savepoints into one savepoint is currently supported by Flink. > > > > However, we're currently working on dynamic scaling which will allow to > adjust the parallelism of your Flink job. This helps you to scale in/out > depending on the workload of your job. However, you would only be able to > scale within a single Flink job and not across Flink jobs. > > > > Cheers, > > Till > > > > On Mon, Aug 1, 2016 at 9:49 PM, Claudia Wegmann > wrote: > > Hey everyone, > > > > I=E2=80=99ve got some questions regarding savepoints in Flink. I have the > following situation: > > > > There is a microservice that reads data from Kafka topics, creates Flink > streams from this data and does different computations/pattern matching > workloads. If the overall workload for this service becomes too big, I wa= nt > to start a new instance of this service and share the work between the > running services. To accomplish that, I thought about using Flinks > savepoint mechanism. But there are some open questions: > > > > 1.) Can I combine two or more savepoints in one program? > Think of two services already running. Now I=E2=80=99m starting up a thir= d > service. The new one would get savepoints from the already running > services. It than would continue computation of some streams while the > other services would discard calculation on these streams now calculated = by > the new service. So, is it possible to combine two or more savepoints in > one program? > > 2.) Another approach I could think of for accomplishing the > introduction of a new service would be, to just take a savepoint of the > streams that change service. Can I only take a savepoint of a part of the > running job? > > Thanks for your comments and best wishes, > > Claudia > > > --94eb2c06fb2ee502800539178002 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Claudia,

1) At the moment the offset= information will be written to the ZooKeeper quorum used by Kafka as well = as to the savepoint. Reading the savepoint is not so easy to do since you w= ould need to know the internal representation of the savepoint. But you cou= ld try to read the Kafka offsets from ZooKeeper.

2= ) That depends a little bit on the deployment and the size of the job. Are = you using a yarn session or a standalone cluster? Then the task manager sho= uld already be registered at the job manager and the deployment for each ta= sk should be in the milli second range. If you start a new yarn application= for a flink job (per job cluster), then it might take a bit longer dependi= ng on how long it takes to allocate the requested resources by Yarn. But on= ce this is done, the deployment for a task should be in the sub second rang= e.

3) If you want to keep the different Flink jobs= separated, then you should submit them separately to a Flink cluster or st= art a Flink cluster per job (e.g. with Yarn). I don't think that this a= bad architecture if you want to fulfil these requirements. However, I'= m not sure whether merging and splitting savepoints will be implemented any= time soon.

Actually we're currently working on= improving Flink's functionality to be started with a dedicated job. Th= is means that you start a job manager which has already the job jar in its = classpath and directly starts executing the contained job. This will be hel= pful for deployment scenarios how they appear when using docker images, for= example. I could imagine that this could be helpful for your use case as w= ell.

Cheers,
Till

On Mon, Aug 1, 2016 at 10:4= 0 PM, Claudia Wegmann <c.wegmann@kasasi.de> wrote:

Hi Till,

=C2=A0

thanks for the quick reply. Too bad,= I thought I was on the right track with savepoints here.

=C2=A0

Some follow-up questions:<= /u>

=C2=A0

1.)=C2=A0=C2=A0=C2=A0 Can I do the whole thing of tra= nsferring state and the position in the Kafka topic manually for one stream= ? In other words: is this information accessible easily?

2.)=C2=A0=C2=A0=C2=A0 In any case I would need to sto= p the running job, change the topology (e.g. the number of streams in the p= rogram) and resume processing. Can you name the overhead of time coming from stopping and sta= rting a Flink job?

3.)=C2=A0=C2=A0=C2=A0 I=E2=80=99m aware of the upcomi= ng feature for scaling in and out. But I don=E2=80=99t quite see, how this = will help me with different services.
I thought of each service having its own Flink instance/cluster. I would co= mmit this service as one job to the dedicated Flink containing all the nece= ssary streams and computations. Is this a bad architecture?
Would it be better to have one big Flink cluster and commit one big Job, wh= ich contains all the streams? (As I got to know, committing multiple jobs t= o one Flink instance is not recommended).
To be honest, I=E2=80=99m not quite there to totally understand the differe= nt deployment options of Flink and how to bring them together with a micros= ervice architecture where I have a service packed as a JAR-File and wanting= to be able to just deploy this JAR-File. I thought of this service containing Flink and then start the JobManager a= nd some TaskManagers from this service and deploy itself as the Flink job w= ith a dedicated entry point. Is this a good idea? Or is it even possible?

=C2=A0

Thanks in advance,

Claudia

=C2=A0

Von: T= ill Rohrmann [mailto:trohrmann@apache.org]
Gesendet: Montag, 1. August 2016 16:21
An: user@= flink.apache.org
Betreff: Re: partial savepoints/combining savepoints<= u>

=C2=A0

Hi Claudia,

=C2=A0

unfortunately neither taking partial savepoints nor = combining multiple savepoints into one savepoint is currently supported by = Flink.

=C2=A0

However, we're currently working on dynamic scal= ing which will allow to adjust the parallelism of your Flink job. This help= s you to scale in/out depending on the workload of your job. However, you w= ould only be able to scale within a single Flink job and not across Flink jobs.

=C2=A0

Cheers,

Till

=C2=A0

On Mon, Aug 1, 2016 at 9:49 PM, Claudia Wegmann <= c.wegmann@kasasi.d= e> wrote:

Hey everyone,

=C2=A0

I=E2=80=99ve got some questions= regarding savepoints in Flink. I have the following situation:

=C2=A0

There is a microservice that re= ads data from Kafka topics, creates Flink streams from this data and does d= ifferent computations/pattern matching workloads. If the overall workload for this service becomes too big, I want to start a n= ew instance of this service and share the work between the running services= . To accomplish that, I thought about using Flinks savepoint mechanism. But= there are some open questions:

=C2=A0

1.)=C2=A0=C2=A0=C2=A0 Can I combine two or more savepoints in one pro= gram?
Think of two services already running. Now I=E2=80=99m starting up a third = service. The new one would get savepoints from the already running services= . It than would continue computation of some streams while the other servic= es would discard calculation on these streams now calculated by the new service. So, is it possible to combine two or mo= re savepoints in one program?

2.)=C2=A0=C2=A0=C2=A0 Another approach I could think of for accomplis= hing the introduction of a new service would be, to just take a savepoint o= f the streams that change service. Can I only take a savepoint of a part of= the running job?

Thanks for your comments and be= st wishes,

Claudia

=C2=A0


--94eb2c06fb2ee502800539178002--