flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: [DISCUSS] State Management API for Streaming
Date Wed, 03 Jun 2015 12:38:44 GMT

Sorry for not responding earlier.

Thanks for taking an initiative there and drafting this. I would like to
have a close look after the 0.9 release is out.

I am in favor of making this a major item for 0.10, to get the state with
all its characteristics defined in a principles fashion: repartitionable,
synchronous vs. asynchronous checkpointing, full vs incremental checkpoints.


On Fri, May 29, 2015 at 2:13 PM, Paris Carbone <parisc@kth.se> wrote:

> Hello fellow squirrels!
> We just made a PR [1] of a prototype targeting flexible state management
> for streaming tasks with the prospect of further implementing on top
> different strategies such as lazy state updates, incremental snapshots and
> state partitioning. You can read more regarding the motivation behind this
> design in the doc Gyula created [2].
> As described in the doc, "managed" operator state and updates are
> explicitly specified by the OperatorState abstraction. Furthermore,
> OperatorState can be partitioned by a key and retrieved by the getState
> method. If the state is partitioned the getState method implicitly returns
> the state of the respective key that is currently processed by the
> operator. It will still be possible to add functionality in order to access
> keys arbitrarily if needed but for now this looks hopefully clear enough.
> Let's discuss here how we can use/modify this api to fit our general needs
> and vision for state management.
> Paris
> [1] https://github.com/apache/flink/pull/747
> [2]
> https://docs.google.com/document/d/1nTn4Tpafsnt-TCT6L1vlHtGGgRevU90yRsUQEmkRMjk
> ?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message