flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4164) Use only one rocksdb column family to store all states in an operator
Date Fri, 08 Jul 2016 12:26:11 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367596#comment-15367596
] 

Aljoscha Krettek commented on FLINK-4164:
-----------------------------------------

I would suggest to copy the existing RocksDBStateBackend and change it to use only one column.
In process of this you might also have to adapt RocksDBReducingState and the other specific
state classes. Once you have that you can test how the new backend behaves.

> Use only one rocksdb column family to store all states in an operator
> ---------------------------------------------------------------------
>
>                 Key: FLINK-4164
>                 URL: https://issues.apache.org/jira/browse/FLINK-4164
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>            Reporter: LiuBiao
>            Priority: Minor
>
> I see now in master branch, there is only one rocksdb instance in an operator, each state
will be assigned into separated column families. It's a good improvement, but I think there
are still some problems. Column families in rocksdb do not share memtable and sst files(correct
me if I am wrong, there is not much docs about column families in rocksdb). If user have thousands
of  states, the overhead will be costly. Memory may will be exhausted, also there will be
too much small sst files.
> If we use prefix in key instead of column family, the memory that rocksdb costed will
be under control. The shortcoming is that we can not optimize states separately, and performance
may be a little worse than before. But I think it is worth to do, it is meaning for some cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message