flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Kloudas <kklou...@gmail.com>
Subject Re: terminate and restart Flink app again, data may lose?
Date Thu, 02 Apr 2020 15:46:45 GMT
Hi Yucai,

How do you terminate the application? Do you take a savepoint before and
you restart from that afterwards?

Also do you mean that the in-progress files do not get deleted after being
committed or that they are not committed?
In the screenshot it seems that only part-0-3 is still not committed.
The rest seem to be finalised.

I assume you are using HDFS, right?

Cheers,
Kostas


On Thu, Apr 2, 2020 at 3:11 PM Yu, Yucai <yyu1@ebay.com> wrote:

> Hi,
>
>
>
> We are using StreamingFileSink in our Flink apps (1.7), we found when
> terminating and restarting app again, a lot of files will in “ingress”
> status, never get a chance to commit.
>
> It may leads to lose data, any idea?
>
>
>
> Thanks,
>
> Lex
>

Mime
View raw message