flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "mingleizhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6849) Refactor operator state backend and internal operator state hierarchy
Date Sat, 10 Jun 2017 08:34:18 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045459#comment-16045459
] 

mingleizhang commented on FLINK-6849:
-------------------------------------

Yep. [~srichter]. We should have a concrete plan refactor operator state backend before we
start do it. About more implementations for that, I think we could choose RocksDB as it's
concrete implementations for it as well and extends an abstract base class that we called
{{AbstractOperatorStateBackend}}. And we can also caches registered state in this abstract
base class. BTW,  If we use RocksDB as its implementations, we can have a large state for
a job. As for the other way to achieve operator state backend, I have no idea about it at
the moment. [~tzulitai] What do you think ? 

> Refactor operator state backend and internal operator state hierarchy
> ---------------------------------------------------------------------
>
>                 Key: FLINK-6849
>                 URL: https://issues.apache.org/jira/browse/FLINK-6849
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>            Reporter: Tzu-Li (Gordon) Tai
>
> Currently, compared to the keyed state backends, the operator state backends, as well
as operator state interfaces, lacks proper hierarchy.
> One issue with this lack of hierarchy is that the general concerns of implementing state
registration is different between the keyed and operator backends (aside from what is naturally
different, such as namespace and key which is not relevant for the operator backend). For
example, in the keyed backend hierarchy, {{AbstractKeyedStateBackend}} has caches that shortcuts
re-accessing already registered state. This behaviour is missing in the operator backend hierarchy,
and for example needs to be explicitly handled by the concrete {{DefaultOperatorStateBackend}}
subclass implementation.
> As of now, the need of a proper hierarchy also on the operator backend side might not
be that prominent, but will mostly likely become more prominent  as we wish to introduce more
state structures for operator state (e.g. a {{MapState}} for operator state has already been
discussed a few times already) as well as more options besides memory-backed operator state.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message