flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rinat <r.shari...@cleverdata.ru>
Subject Re: State life-cycle for different state-backend implementations
Date Wed, 13 Jun 2018 07:02:44 GMT
Hi Sihua, Thx for your reply

> On 9 Jun 2018, at 11:42, sihua zhou <summerleafs@163.com> wrote:
> 
> Hi Rinat,
> 
> I think there is one configuration {{state.checkpoints.num-retained}} to control the
maximum number of completed checkpoints to retain, the default value is 1. So the risk you
mentioned should not happen. Refer to https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#checkpointing
<https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#checkpointing>
you could find more configurations of checkpoint.    
> 
> Best, Sihua
> 
> 
> On 06/8/2018 22:55,Rinat<r.sharipov@cleverdata.ru> <mailto:r.sharipov@cleverdata.ru>
wrote: 
> Hi mates, got a question about different state backends.
> 
> As I've properly understood, on every checkpoint, Flink flushes it’s current state
into backend. In case of FsStateBackend we’ll have a separate file for each checkpoint,
and during the job lifecycle we got a risk of 
> a huge amount of state files in hdfs, that is not very cool for a hadoop name-node.
> 
> Does Flink have any clean-up strategies for it’s state in different implementation
of backends ? If you could provide any links, where I could read about more details of this
process, it’ll be awesome ))
> 
> Thx a lot for your help.
> 
> Sincerely yours,
> Rinat Sharipov
> Software Engineer at 1DMP CORE Team
> 
> email: r.sharipov@cleverdata.ru <mailto:a.totmakov@cleverdata.ru>
> mobile: +7 (925) 416-37-26
> 
> CleverDATA
> make your data clever
> 
> 

Sincerely yours,
Rinat Sharipov
Software Engineer at 1DMP CORE Team

email: r.sharipov@cleverdata.ru <mailto:a.totmakov@cleverdata.ru>
mobile: +7 (925) 416-37-26

CleverDATA
make your data clever


Mime
View raw message