flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paris Carbone <par...@kth.se>
Subject [DISCUSS] State Management API for Streaming
Date Fri, 29 May 2015 12:13:29 GMT
Hello fellow squirrels!


We just made a PR [1] of a prototype targeting flexible state management for streaming tasks
with the prospect of further implementing on top different strategies such as lazy state updates,
incremental snapshots and state partitioning. You can read more regarding the motivation behind
this design in the doc Gyula created [2].


As described in the doc, "managed" operator state and updates are explicitly specified by
the OperatorState abstraction. Furthermore, OperatorState can be partitioned by a key and
retrieved by the getState method. If the state is partitioned the getState method implicitly
returns the state of the respective key that is currently processed by the operator. It will
still be possible to add functionality in order to access keys arbitrarily if needed but for
now this looks hopefully clear enough.


Let's discuss here how we can use/modify this api to fit our general needs and vision for
state management.


Paris


[1] https://github.com/apache/flink/pull/747

[2] https://docs.google.com/document/d/1nTn4Tpafsnt-TCT6L1vlHtGGgRevU90yRsUQEmkRMjk

?


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message