flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-5590) Create a proper internal state hierarchy
Date Fri, 20 Jan 2017 13:56:26 GMT
Stephan Ewen created FLINK-5590:

             Summary: Create a proper internal state hierarchy
                 Key: FLINK-5590
                 URL: https://issues.apache.org/jira/browse/FLINK-5590
             Project: Flink
          Issue Type: Improvement
          Components: State Backends, Checkpointing
    Affects Versions: 1.2.0
            Reporter: Stephan Ewen
            Assignee: Stephan Ewen
             Fix For: 1.3.0

Currently, the state interfaces (like {{ListState}}, {{ValueState}}, {{ReducingState}}) are
very sparse and contain only methods exposed to the users. That makes sense to keep the public
stable API minimal

At the same time, the runtime needs more methods for its internal interaction with state,
such as:
  - setting namespaces
  - accessing raw values
  - merging namespaces

These are currently realized by re-creating or re-obtaining the state objects from the KeyedStateBackend.
That method causes quite an overhead for each access to the state

The KeyedStateBackend tries to do some tricks to reduce that overhead, but does it only partially
and induces other overhead in the course.

The root cause of all these issues is a problem in the design: There is no proper "internal
state abstraction" in a similar way as there is an external state abstraction (the public
state API).

We should add a similar hierarchy of states for the internal methods. It would look like in
the example below:

 *             State
 *               |
 *               +-------------------InternalKvState
 *               |                         |
 *          MergingState                   |
 *               |                         |
 *               +-----------------InternalMergingState
 *               |                         |
 *      +--------+------+                  |
 *      |               |                  |
 * ReducingState    ListState        +-----+-----------------+
 *      |               |            |                       |
 *      +-----------+   +-----------   -----------------InternalListState
 *                  |                |
 *                  +---------InternalReducingState

This message was sent by Atlassian JIRA

View raw message