flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Ruppert <ste...@fullcontact.com>
Subject Flink streaming with 1+ TB of managed state
Date Sat, 19 Nov 2016 00:41:35 GMT
Hi,

Is anybody currently running flink streaming with north of a terabyte
(TB) of managed state? If you are, can you share your experiences wrt
hardware, tuning, recovery situations, etc?

I'm evaluating flink for a use case I estimate will take around 5TB of
state in total, but looking at the actual implementation of the
rocksDB state and current lack of incremental checkpointing or
recovery, it doesn't seem feasible.

I have successfully tested flink up to roughly 90GB of managed state
in rocksDB, but that's taking 5 minutes to checkpoint or recover (on a
pretty beefy YARN cluster).

For most cases, my state updates are idempotent and can be moved to
something external. However, it'd be nice to know of any current of
future plans for running flink at the terabyte scale.

--Steven

-- 
*CONFIDENTIALITY NOTICE: This email message, and any documents, files or 
previous e-mail messages attached to it is for the sole use of the intended 
recipient(s) and may contain confidential and privileged information. Any 
unauthorized review, use, disclosure or distribution is prohibited. If you 
are not the intended recipient, please contact the sender by reply email 
and destroy all copies of the original message.*

Mime
View raw message