Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 873E2200B51 for ; Mon, 1 Aug 2016 16:20:58 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 85C01160A6C; Mon, 1 Aug 2016 14:20:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CB2A1160A66 for ; Mon, 1 Aug 2016 16:20:57 +0200 (CEST) Received: (qmail 12531 invoked by uid 500); 1 Aug 2016 14:20:56 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 12522 invoked by uid 99); 1 Aug 2016 14:20:56 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Aug 2016 14:20:56 +0000 Received: from mail-qk0-f178.google.com (mail-qk0-f178.google.com [209.85.220.178]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 813691A0118 for ; Mon, 1 Aug 2016 14:20:56 +0000 (UTC) Received: by mail-qk0-f178.google.com with SMTP id s63so146739336qkb.2 for ; Mon, 01 Aug 2016 07:20:56 -0700 (PDT) X-Gm-Message-State: AEkoouswSho0DM+bTx4GPtzK5tXJ4T6vpHbimGRcw0YunlGJBH1vpL7i7gD49a2UCkz5siyQKTs6as8txRaZug== X-Received: by 10.55.179.129 with SMTP id c123mr71384849qkf.50.1470061255430; Mon, 01 Aug 2016 07:20:55 -0700 (PDT) MIME-Version: 1.0 Received: by 10.55.10.136 with HTTP; Mon, 1 Aug 2016 07:20:54 -0700 (PDT) In-Reply-To: <9633659C9F9D924DB69F4F53CDF42743BD1CFE@kasasiserver> References: <9633659C9F9D924DB69F4F53CDF42743BD1CFE@kasasiserver> From: Till Rohrmann Date: Mon, 1 Aug 2016 22:20:54 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: partial savepoints/combining savepoints To: user@flink.apache.org Content-Type: multipart/alternative; boundary=94eb2c0650eaedbb830539034e35 archived-at: Mon, 01 Aug 2016 14:20:58 -0000 --94eb2c0650eaedbb830539034e35 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Claudia, unfortunately neither taking partial savepoints nor combining multiple savepoints into one savepoint is currently supported by Flink. However, we're currently working on dynamic scaling which will allow to adjust the parallelism of your Flink job. This helps you to scale in/out depending on the workload of your job. However, you would only be able to scale within a single Flink job and not across Flink jobs. Cheers, Till On Mon, Aug 1, 2016 at 9:49 PM, Claudia Wegmann wrote= : > Hey everyone, > > > > I=E2=80=99ve got some questions regarding savepoints in Flink. I have the > following situation: > > > > There is a microservice that reads data from Kafka topics, creates Flink > streams from this data and does different computations/pattern matching > workloads. If the overall workload for this service becomes too big, I wa= nt > to start a new instance of this service and share the work between the > running services. To accomplish that, I thought about using Flinks > savepoint mechanism. But there are some open questions: > > > > 1.) Can I combine two or more savepoints in one program? > Think of two services already running. Now I=E2=80=99m starting up a thir= d > service. The new one would get savepoints from the already running > services. It than would continue computation of some streams while the > other services would discard calculation on these streams now calculated = by > the new service. So, is it possible to combine two or more savepoints in > one program? > > 2.) Another approach I could think of for accomplishing the > introduction of a new service would be, to just take a savepoint of the > streams that change service. Can I only take a savepoint of a part of the > running job? > > Thanks for your comments and best wishes, > > Claudia > --94eb2c0650eaedbb830539034e35 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Claudia,

unfortunately neither takin= g partial savepoints nor combining multiple savepoints into one savepoint i= s currently supported by Flink.

However, we're= currently working on dynamic scaling which will allow to adjust the parall= elism of your Flink job. This helps you to scale in/out depending on the wo= rkload of your job. However, you would only be able to scale within a singl= e Flink job and not across Flink jobs.

Cheers,
Till

On Mon, Aug 1, 2016 at 9:49 PM, Claudia Wegmann <= ;c.wegmann@kasasi.= de> wrote:

Hey everyone,

=C2=A0

I=E2=80=99ve got some questions= regarding savepoints in Flink. I have the following situation:

=C2=A0

There is a microservice that re= ads data from Kafka topics, creates Flink streams from this data and does d= ifferent computations/pattern matching workloads. If the overall workload f= or this service becomes too big, I want to start a new instance of this service and share the work between the run= ning services. To accomplish that, I thought about using Flinks savepoint m= echanism. But there are some open questions:

=C2=A0

1.)=C2=A0=C2=A0=C2=A0 Can I combine two or more = savepoints in one program?
Think of two services already running. Now I=E2=80=99m starting up a third = service. The new one would get savepoints from the already running services= . It than would continue computation of some streams while the other servic= es would discard calculation on these streams now calculated by the new service. So, is it possible to combine two or mo= re savepoints in one program?

2.)=C2=A0=C2=A0=C2=A0 Another approach I could t= hink of for accomplishing the introduction of a new service would be, to ju= st take a savepoint of the streams that change service. Can I only take a s= avepoint of a part of the running job?

Thanks for your comments and be= st wishes,

Claudia


--94eb2c0650eaedbb830539034e35--