flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: partial savepoints/combining savepoints
Date Mon, 01 Aug 2016 14:20:54 GMT
Hi Claudia,

unfortunately neither taking partial savepoints nor combining multiple
savepoints into one savepoint is currently supported by Flink.

However, we're currently working on dynamic scaling which will allow to
adjust the parallelism of your Flink job. This helps you to scale in/out
depending on the workload of your job. However, you would only be able to
scale within a single Flink job and not across Flink jobs.


On Mon, Aug 1, 2016 at 9:49 PM, Claudia Wegmann <c.wegmann@kasasi.de> wrote:

> Hey everyone,
> I’ve got some questions regarding savepoints in Flink. I have the
> following situation:
> There is a microservice that reads data from Kafka topics, creates Flink
> streams from this data and does different computations/pattern matching
> workloads. If the overall workload for this service becomes too big, I want
> to start a new instance of this service and share the work between the
> running services. To accomplish that, I thought about using Flinks
> savepoint mechanism. But there are some open questions:
> 1.)    Can I combine two or more savepoints in one program?
> Think of two services already running. Now I’m starting up a third
> service. The new one would get savepoints from the already running
> services. It than would continue computation of some streams while the
> other services would discard calculation on these streams now calculated by
> the new service. So, is it possible to combine two or more savepoints in
> one program?
> 2.)    Another approach I could think of for accomplishing the
> introduction of a new service would be, to just take a savepoint of the
> streams that change service. Can I only take a savepoint of a part of the
> running job?
> Thanks for your comments and best wishes,
> Claudia

View raw message