flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Ying (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-12005) [State TTL] Event time support
Date Mon, 01 Jun 2020 02:15:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-12005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120701#comment-17120701

Li Ying commented on FLINK-12005:

Hi [~liyu], I'm wondering when this feature will be finished?  It's really useful and necessary
in some cases.

> [State TTL] Event time support
> ------------------------------
>                 Key: FLINK-12005
>                 URL: https://issues.apache.org/jira/browse/FLINK-12005
>             Project: Flink
>          Issue Type: New Feature
>          Components: API / DataStream, Runtime / State Backends
>            Reporter: Andrey Zagrebin
>            Assignee: Yu Li
>            Priority: Major
> The event time is opted for in StateTtlConfig by setting TtlTimeCharacteristic.EventTime.
> To enable event time support, the updated watermark needs to be passed to the state backend,
shared with TTL state wrappers and additional cleanup strategies (snapshot transformers and
compaction filter).
> h3. Event time provider
> Additional implementation of TtlTimeProvider, which holds current watermark, needs to
be passed to the state backend at the moment of its creation in StreamTaskStateInitializerImpl.
There several ways to update watermark in this implementation of TtlTimeProvider:
>  * in InternalTimeServiceManager.advanceWatermark explicitly
>  * InternalTimeServiceManager/InternalTimerServiceImpl could be refactored to use shared
EventTimeService which holds current updatable watermark and wrapped by TtlTimeProvider
> The TTL state wrapping factory should create TTL state wrappers and snapshot transformers
with TtlTimeProvider selected by TtlTimeCharacteristic.
> h3. RocksDB TTL compaction filter
> The RocksDB TTL compaction filter factory needs to get selected TtlTimeProvider when
it gets configured. There are two ways:
>  * make it volatile and settable in RocksDbTtlCompactFiltersManager.TimeProviderWrapper,
track it in RocksDbTtlCompactFiltersManager along with FlinkCompactionFilterFactory to configure
later before configuring FlinkCompactionFilterFactory.
>  * Move FlinkCompactionFilter.TimeProvider from FlinkCompactionFilterFactory to ConfigHolder
and set selected TtlTimeProvider with the Config.
> The second option does not use volatile variable and should be more performant but needs
changing RocksDB java client and either releasing new version FRocksDB or Flink RocksDB extensions.

This message was sent by Atlassian Jira

View raw message