flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9514) Create wrapper with TTL logic for value state
Date Fri, 22 Jun 2018 16:11:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520540#comment-16520540
] 

ASF GitHub Bot commented on FLINK-9514:
---------------------------------------

Github user StefanRRichter commented on a diff in the pull request:

    https://github.com/apache/flink/pull/6186#discussion_r197494851
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/ttl/TtlMapState.java
---
    @@ -0,0 +1,132 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.runtime.state.ttl;
    +
    +import org.apache.flink.api.common.typeutils.TypeSerializer;
    +import org.apache.flink.runtime.state.internal.InternalMapState;
    +import org.apache.flink.util.FlinkRuntimeException;
    +
    +import java.util.AbstractMap;
    +import java.util.Collections;
    +import java.util.Iterator;
    +import java.util.Map;
    +import java.util.stream.Collectors;
    +import java.util.stream.Stream;
    +import java.util.stream.StreamSupport;
    +
    +/**
    + * This class wraps map state with TTL logic.
    + *
    + * @param <K> The type of key the state is associated to
    + * @param <N> The type of the namespace
    + * @param <UK> Type of the user entry key of state with TTL
    + * @param <UV> Type of the user entry value of state with TTL
    + */
    +class TtlMapState<K, N, UK, UV>
    +	extends AbstractTtlState<K, N, Map<UK, UV>, Map<UK, TtlValue<UV>>,
InternalMapState<K, N, UK, TtlValue<UV>>>
    +	implements InternalMapState<K, N, UK, UV> {
    +	TtlMapState(
    +		InternalMapState<K, N, UK, TtlValue<UV>> original,
    +		TtlConfig config,
    +		TtlTimeProvider timeProvider,
    +		TypeSerializer<Map<UK, UV>> valueSerializer) {
    +		super(original, config, timeProvider, valueSerializer);
    +	}
    +
    +	@Override
    +	public UV get(UK key) throws Exception {
    +		return getWithTtlCheckAndUpdate(() -> original.get(key), v -> original.put(key,
v), () -> original.remove(key));
    +	}
    +
    +	@Override
    +	public void put(UK key, UV value) throws Exception {
    +		original.put(key, wrapWithTs(value));
    +	}
    +
    +	@Override
    +	public void putAll(Map<UK, UV> map) throws Exception {
    +		if (map == null) {
    +			return;
    +		}
    +		Map<UK, TtlValue<UV>> ttlMap = map.entrySet().stream()
    +			.collect(Collectors.toMap(Map.Entry::getKey, e -> wrapWithTs(e.getValue())));
    +		original.putAll(ttlMap);
    +	}
    +
    +	@Override
    +	public void remove(UK key) throws Exception {
    +		original.remove(key);
    +	}
    +
    +	@Override
    +	public boolean contains(UK key) throws Exception {
    +		return get(key) != null;
    +	}
    +
    +	@Override
    +	public Iterable<Map.Entry<UK, UV>> entries() throws Exception {
    +		return entriesStream()::iterator;
    +	}
    +
    +	private Stream<Map.Entry<UK, UV>> entriesStream() throws Exception {
    +		Iterable<Map.Entry<UK, TtlValue<UV>>> withTs = original.entries();
    +		withTs = withTs == null ? Collections.emptyList() : withTs;
    +		return StreamSupport
    +			.stream(withTs.spliterator(), false)
    --- End diff --
    
    I saw that some of the classes make heavy use of streams by getting spliterators from
collections. While the code is concise, this creates some default adapter from iterator to
spliterator under the hood. IIRC this can have significant performance impact, especially
if used with hot code paths. State access in Flink can be considered a hot path for some cases.
It is hard to quantify the impact just from looking at it, but when using this kind of api
adapters in "low-level" classes, please be aware of the potential impact. We might want to
have a look at the performance and tune if needed. Should be ok for now because there is no
regression in existing code, but we might want to measure this for heap based state eventually.


> Create wrapper with TTL logic for value state
> ---------------------------------------------
>
>                 Key: FLINK-9514
>                 URL: https://issues.apache.org/jira/browse/FLINK-9514
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.6.0
>            Reporter: Andrey Zagrebin
>            Assignee: Andrey Zagrebin
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.6.0
>
>
> TTL state decorator uses original state with packed TTL and add TTL logic using time
provider:
> {code:java}
> TtlValueState<V> implements ValueState<V> {
>   ValueState<TtlValue<V>> underlyingState;
>   InternalTimeService timeProvider;
>   V value() {
>     TtlValue<V> valueWithTtl = underlyingState.get();
>     // ttl logic here (e.g. update timestamp)
>     return valueWithTtl.getValue();
>   }
>   void update() { ... underlyingState.update(valueWithTtl) ...  }
> }
> {code}
> TTL decorators are apply to state produced by normal state binder in its TTL wrapper
from FLINK-9513



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message