flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juha Mynttinen <juha.myntti...@king.com>
Subject Performance issue associated with managed RocksDB memory
Date Wed, 24 Jun 2020 12:24:28 GMT
Hello there,

In Flink 1.10 the configuration parameter state.backend.rocksdb.memory.managed defaults to
true. This is great, since it makes it simpler to stay within the memory budget e.g. when
running in a container environment. However, I've noticed performance issues when the switch
is enabled.

To isolate where the issue is I've written some test code.

If I create a Flink job that has a single "heavy" operator (call it X) that just keeps a simple
state (per user) things work fast when testing how many events / s sec the job can process.
However, If I add downstream of X a simplest possible window operator, things can get slow,
especially when I increase the parallelism. With slow I mean even 90% less events / second.
The bad thing is that things get slower when parallelism is increased.

What seems to happen is that the window operator constantly flushes memtable(s) because X
fills up the shared ḿemtable memory. This naturally causes the window operator also to compact
its RocksDB db. I can see the constant flush / compaction in RocksDB log and in the fact that
there are new SST files all the time. This flushing is (theoretically) unneeded, since the
size of the state is < 1kb and it really should fit to the memtable.

If a disable the managed memory switch, things are fast (even if I increase parallelism).
There are magnitudes less flushes and compactions, I assume because now the state fits nicely
to the memtable. Also, if I downgrade to Flink 1.9, things are fast (there's no shared memory

I have a tester program that clearly illustrates the issue and tests results in a Google Sheet.
The tester is too complex to be included inline here. Should I file a JIRA ticket or where
should I put the test code?

View raw message