Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Sat, 10 Jun 2017 09:49:19 +0000 (UTC)
From: "mingleizhang (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.13077205.1496657275000.45674.1497088159001@Atlassian.JIRA>
In-Reply-To: <JIRA.13077205.1496657275000@Atlassian.JIRA>
References: <JIRA.13077205.1496657275000@Atlassian.JIRA> <JIRA.13077205.1496657275522@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (FLINK-6849) Refactor operator state
 backend and internal operator state hierarchy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 10 Jun 2017 09:49:26 -0000


    [ https://issues.apache.org/jira/browse/FLINK-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045475#comment-16045475 ] 

mingleizhang edited comment on FLINK-6849 at 6/10/17 9:48 AM:
--------------------------------------------------------------

[~tzulitai] [~srichter] I just do a little refactor for the {{OperatorStateBackend}} interface by adding two method. And below is it's the first version sample code. Please helps to reiview and provide some suggestions for this change. Thanks. About what is relationship of opeartor state and keyed state. We can think it it just a partitioned operator state. :)

{code}/**
	 * Creates or retrieves a operator state backed by this state backend.
	 *
	 * @param namespaceSerializer The serializer used for the namespace type of the state.
	 * @param stateDescriptor The identifier for the state. This contains name and can create a default state value.
	 *
	 * @param <N> The type of the namespace.
	 * @param <S> The type of the state.
	 *
	 * @return A new operator state backed by this backend.
	 *
	 * @throws Exception Exceptions may occur during initialization of the state and should be forwarded.
	 */
	<N, S extends State, T> S getOrCreateOperatorState(
			TypeSerializer<N> namespaceSerializer,
			StateDescriptor<S, T> stateDescriptor) throws Exception;

	/**
	 * Creates or retrieves a operator state backed by this backend.
	 *
	 * TODO: NOTE: This method does a lot of work caching / retrieving states just to update the namespace.
	 * 		 This method should be removed for the sake of namespaces being lazily fetched from the operator
	 * 		 state backend, or being set on the state directly.
	 *
	 * @param stateDescriptor The identifier for the state. This contains name and can create a default state value.
	 *
	 * @param <N> The type of the namespace.
	 * @param <S> The type of the state.
	 *
	 * @return a new operator state backed by this backend.
	 *
	 * @throws Exception Exceptions may occur during initialization of the state and should be forwarded.
	 */
	<N, S extends State> S getOperatorState(
			N namespace,
			TypeSerializer<N> namespaceSerializer,
			StateDescriptor<S, ?> stateDescriptor) throws Exception;{code}


was (Author: mingleizhang):
[~tzulitai] [~srichter] I just do a little refactor for the {{OperatorStateBackend}} interface by adding two method. And below is it's the first version sample code. Please helps to reiview and provide some suggestions for this change. Thanks.

{code}/**
	 * Creates or retrieves a operator state backed by this state backend.
	 *
	 * @param namespaceSerializer The serializer used for the namespace type of the state.
	 * @param stateDescriptor The identifier for the state. This contains name and can create a default state value.
	 *
	 * @param <N> The type of the namespace.
	 * @param <S> The type of the state.
	 *
	 * @return A new operator state backed by this backend.
	 *
	 * @throws Exception Exceptions may occur during initialization of the state and should be forwarded.
	 */
	<N, S extends State, T> S getOrCreateOperatorState(
			TypeSerializer<N> namespaceSerializer,
			StateDescriptor<S, T> stateDescriptor) throws Exception;

	/**
	 * Creates or retrieves a operator state backed by this backend.
	 *
	 * TODO: NOTE: This method does a lot of work caching / retrieving states just to update the namespace.
	 * 		 This method should be removed for the sake of namespaces being lazily fetched from the operator
	 * 		 state backend, or being set on the state directly.
	 *
	 * @param stateDescriptor The identifier for the state. This contains name and can create a default state value.
	 *
	 * @param <N> The type of the namespace.
	 * @param <S> The type of the state.
	 *
	 * @return a new operator state backed by this backend.
	 *
	 * @throws Exception Exceptions may occur during initialization of the state and should be forwarded.
	 */
	<N, S extends State> S getOperatorState(
			N namespace,
			TypeSerializer<N> namespaceSerializer,
			StateDescriptor<S, ?> stateDescriptor) throws Exception;{code}

> Refactor operator state backend and internal operator state hierarchy
> ---------------------------------------------------------------------
>
>                 Key: FLINK-6849
>                 URL: https://issues.apache.org/jira/browse/FLINK-6849
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>            Reporter: Tzu-Li (Gordon) Tai
>
> Currently, compared to the keyed state backends, the operator state backends, as well as operator state interfaces, lacks proper hierarchy.
> One issue with this lack of hierarchy is that the general concerns of implementing state registration is different between the keyed and operator backends (aside from what is naturally different, such as namespace and key which is not relevant for the operator backend). For example, in the keyed backend hierarchy, {{AbstractKeyedStateBackend}} has caches that shortcuts re-accessing already registered state. This behaviour is missing in the operator backend hierarchy, and for example needs to be explicitly handled by the concrete {{DefaultOperatorStateBackend}} subclass implementation.
> As of now, the need of a proper hierarchy also on the operator backend side might not be that prominent, but will mostly likely become more prominent  as we wish to introduce more state structures for operator state (e.g. a {{MapState}} for operator state has already been discussed a few times already) as well as more options besides memory-backed operator state.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)