flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Flink streaming with 1+ TB of managed state
Date Mon, 21 Nov 2016 14:47:12 GMT
Some background in the Incremental Checkpointing: It is not in the system,
but we have a quite advanced design and some committers/contributors are
currently starting the effort.

My personal estimate is that it would be available in some months (Q1 next


On Sat, Nov 19, 2016 at 4:07 PM, Gyula Fóra <gyula.fora@gmail.com> wrote:

> Hi Steven,
> As Robert said some of our jobs have state sizes around a TB or more. We
> use the RocksDB state backend with some configs tuned to perform well on
> SSDs (you can get some tips here: https://www.youtube.com/watch?
> v=pvUqbIeoPzM).
> We checkpoint our state to Ceph (similar to HDFS but this is what we have
> :)), and it takes 15-25 minutes for the larger jobs to perform the
> checkpoints/restore. As this runs async in the background it doesnt hurt
> our runtime performance, the only problems are with the strain on the
> network sometimes especially when many jobs are restored at the same time.
> Incremental checkpoints would definitely be crazy useful in our case as
> only a very small percentage of our state is updated between snapshots but
> it is still feasible as it is for now.
> Let me know if I can help with any details.
> Cheers,
> Gyula
> Robert Metzger <rmetzger@apache.org> ezt írta (időpont: 2016. nov. 19.,
> Szo, 13:16):
>> Hi Steven,
>> According to this presentation, King.com is using Flink with terabytes of
>> state: http://flink-forward.org/wp-content/uploads/2016/
>> 07/Gyulo-Fo%CC%81ra-RBEA-Scalable-Real-Time-Analytics-
>> at-King.compressed.pdf (see Page 4 specifically)
>> For the 90GB experiment, what is the expected time for transferring 90 GB
>> of data in your environment?
>> Regards,
>> Robert
>> On Sat, Nov 19, 2016 at 1:41 AM, Steven Ruppert <steven@fullcontact.com>
>> wrote:
>> Hi,
>> Is anybody currently running flink streaming with north of a terabyte
>> (TB) of managed state? If you are, can you share your experiences wrt
>> hardware, tuning, recovery situations, etc?
>> I'm evaluating flink for a use case I estimate will take around 5TB of
>> state in total, but looking at the actual implementation of the
>> rocksDB state and current lack of incremental checkpointing or
>> recovery, it doesn't seem feasible.
>> I have successfully tested flink up to roughly 90GB of managed state
>> in rocksDB, but that's taking 5 minutes to checkpoint or recover (on a
>> pretty beefy YARN cluster).
>> For most cases, my state updates are idempotent and can be moved to
>> something external. However, it'd be nice to know of any current of
>> future plans for running flink at the terabyte scale.
>> --Steven
>> --
>> *CONFIDENTIALITY NOTICE: This email message, and any documents, files or
>> previous e-mail messages attached to it is for the sole use of the
>> intended
>> recipient(s) and may contain confidential and privileged information. Any
>> unauthorized review, use, disclosure or distribution is prohibited. If you
>> are not the intended recipient, please contact the sender by reply email
>> and destroy all copies of the original message.*

View raw message