flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Qin <qinnc...@gmail.com>
Subject Re: Data loss in Flink Kafka Pipeline
Date Thu, 07 Dec 2017 18:13:31 GMT
Nishu

You might consider sideouput with metrics at least after window. I would
suggest having that to catch data screw or partition screw in all flink
jobs  and amend if needed.

Chen

On Thu, Dec 7, 2017 at 9:48 AM Fabian Hueske <fhueske@gmail.com> wrote:

> Is it possible that the data is dropped due to being late, i.e., records
> with timestamps behind the current watemark?
> What kind of operations does your program consist of?
>
> Best, Fabian
>
> 2017-12-07 10:20 GMT+01:00 Sendoh <unicorn.banachi@gmail.com>:
>
>> I would recommend to also print the count of input and output of each
>> operator by using Accumulator.
>>
>> Cheers,
>>
>> Sendoh
>>
>>
>>
>> --
>> Sent from:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>
>
> --
Chen
Software Eng, Facebook

Mime
View raw message