Hi all!

I would suggest a change of the current default for timers. A bit of background:

  - Timers (for windows, process functions, etc.) are state that is managed and checkpointed as well.
  - When using the MemoryStateBackend and the FsStateBackend, timers are kept on the JVM heap, like regular state.
  - When using the RocksDBStateBackend, timers can be kept in RocksDB (like other state) or on the JVM heap. The JVM heap is the default though!

I find this a bit un-intuitive and would propose to change this to let the RocksDBStateBackend store all state in RocksDB by default.
The rationale being that if there is a tradeoff (like here), safe and scalable should be the default and unsafe performance be an explicit choice.

This sentiment seems to be shared by various users as well, see https://twitter.com/StephanEwen/status/1214590846168903680 and https://twitter.com/StephanEwen/status/1214594273565388801
We would of course keep the switch and mention in the performance tuning section that this is an option.

# RocksDB State Backend Timers on Heap
  - Pro: faster
  - Con: not memory safe, GC overhead, longer synchronous checkpoint time, no incremental checkpoints

#  RocksDB State Backend Timers on in RocksDB
  - Pro: safe and scalable, asynchronously and incrementally checkpointed
  - Con: performance overhead.

Please chime in and let me know what you think.

Best,
Stephan