flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sihua zhou" <summerle...@163.com>
Subject Re:State life-cycle for different state-backend implementations
Date Sat, 09 Jun 2018 08:42:16 GMT
Hi Rinat,

I think there is one configuration {{state.checkpoints.num-retained}} to control the maximum
number of completed checkpoints to retain, the default value is 1. So the risk you mentioned
should not happen. Refer to https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#checkpointing
you could find more configurations of checkpoint.    

Best, Sihua

On 06/8/2018 22:55,Rinat<r.sharipov@cleverdata.ru> wrote:
Hi mates, got a question about different state backends.

As I've properly understood, on every checkpoint, Flink flushes it’s current state into
backend. In case of FsStateBackend we’ll have a separate file for each checkpoint, and during
the job lifecycle we got a risk of 
a huge amount of state files in hdfs, that is not very cool for a hadoop name-node.

Does Flink have any clean-up strategies for it’s state in different implementation of backends
? If you could provide any links, where I could read about more details of this process, it’ll
be awesome ))

Thx a lot for your help.

Sincerely yours,

Rinat Sharipov
Software Engineer at 1DMP CORE Team

email: r.sharipov@cleverdata.ru
mobile: +7 (925) 416-37-26

make your data clever

View raw message