flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3947) Provide low level access to RocksDB state backend
Date Wed, 25 May 2016 10:29:12 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299827#comment-15299827

Stephan Ewen commented on FLINK-3947:

Concerning iterating over an ordered {{MapState}}: Right now we explicitly limit the access
to keys other than the element's current key.
The reason is that keys may be redistributed as part of the dynamic scaling feature: https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9WyssQ63w/edit

Can you explain a bit what you would need the ordered map state for? Maybe we can find a different
way to incorporate that behavior, while staying compatible with the key group and rescaling

> Provide low level access to RocksDB state backend
> -------------------------------------------------
>                 Key: FLINK-3947
>                 URL: https://issues.apache.org/jira/browse/FLINK-3947
>             Project: Flink
>          Issue Type: Improvement
>          Components: state backends
>    Affects Versions: 1.0.3
>            Reporter: Elias Levy
> The current state API is limiting and some implementations are not as efficient as they
could be, particularly when working with large states. For instance, a ListState is append
only.  You cannot remove values from the list.  And the RocksDBListState get() implementation
reads all list values from RocksDB instead of returning an Iterable that only reads values
as needed.
> Furthermore, RocksDB is an ordered KV store, yet there is no ordered map state API with
an ability to iterate over the stored values in order.

This message was sent by Atlassian JIRA

View raw message