flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyf...@apache.org>
Subject [DISCUSSION] Database state backend
Date Mon, 14 Mar 2016 10:20:25 GMT
Hello everyone!

I would like to start a discussion regarding the future of the Database
state backend for the streaming API. The main question is whether we want
to keep this as a flink-contrib moduel and continue development as part of
Flink or should we move it to an outside library.

Just as a quick refresher, the DB state backend is designed to store
streaming state in JDBC supporting databases (the connector in the flink
repo supports MySQL). There is an adapter interface that can be implemented
to use different databases other than MySQL.

Advantages:

   - Can store huge states (tested with more than a TB of state with MySQL)
   - Fast, incremental checkpoints by default
   - In-memory cache for hot keys
   - Supports rescaling easily (restarting the job with different
   parallelism would still work as expected)
   - Supports database sharding with custom data partitioner
   - Exposes an easily accessible state view to the outside (will contain
   uncheckpointed state)

Disadvantages:

   - Lookup/Update might be much slower than local state backend (RocksDB
   for instance)
   - Incremental checkpoints are not very well supported with the current
   runtime so state cleanup/compaction does not happen automatically
   - Restore speed depends on database indexing as it involves range
   deletes based on timestamps (this might get slow on very large states)
   - The caching layer is not memory aware, this should be moved to flink
   managed memory

We developed (and contributed) the DB state backend to be able to handle
large state sizes, but have moved to RocksDB for much better read/write
performance. I have a pending PR with some improvements to the MySQL
adapter but the question is whether we should keep this as part of Flink
contrib or move it to an external library.

I am personally undecided whether there are strong use cases that would
favour this backend over RocksDB.

Any opinions?

Cheers,
Gyula

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message