flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prashant Nayak <prash...@intellifylearning.com>
Subject S3 recovery and checkpoint directories exhibit explosive growth
Date Thu, 13 Jul 2017 20:52:01 GMT
We’re using Flink 1.3.1 on Mesos, with HA/recovery stored in S3 using
RocksDB with incremental checkpointing.

We have enabled external checkpoints (every 30s), retaining the two latest
external checkpoints.

We are trying to track down something we see happening where the recovery,
checkpoint and external checkpoints directories literally explode in size.
  When we have about 20-30 jobs running, we see these directories grow from
60-70 files to hundreds of thousands of files.


For e.g.  here are stats on a recent state we got to

flink/checkpoints    774824 Objects 5.4 GB
flink/ext-checkpoints 229300 Objects 171.8 MB
flink/recovery 229531 Objects - 909.7MB

Under these circumstances, the JobManager/AppMaster becomes completely
unresponsive.  The only option seems to be to stop it, cull those
directories and restart, which is not ideal.

Appreciate any insight on this…

Also, is there any documentation on best/ops practices on maintaining these
directories? For e.g. is it OK  to have a reaper script that cleans out
these directories for everything that is say 3 days old?

Thanks
Prashant

Mime
View raw message