flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kien Truong <duckientru...@gmail.com>
Subject Re: High back-pressure after recovering from a save point
Date Thu, 13 Jul 2017 23:15:52 GMT
Hi Fabian, 
This happens to me even when the restore is immediate, so there's not much data in Kafka to
catch up (5 minutes max) 

Regards 
Kien

On Jul 13, 2017, 23:40, at 23:40, Fabian Hueske <fhueske@gmail.com> wrote:
>I would guess that this is quite usual because the job has to
>"catch-up"
>work.
>For example, if you took a save point two days ago and restore the job
>today, the input data of the last two days has been written to Kafka
>(assuming Kafka as source) and needs to be processed.
>The job will now read as fast as possible from Kafka to catch-up to the
>presence. This means the data is much fast ingested (as fast as Kafka
>can
>read and ship it) than during regular processing (as fast as your
>sources
>produce).
>The processing speed is bound by your Flink job which means there will
>be
>backpressure.
>
>Once the job caught-up, the backpressure should disappear.
>
>Best, Fabian
>
>2017-07-13 15:48 GMT+02:00 Kien Truong <duckientruong@gmail.com>:
>
>> Hi all,
>>
>> I have one job where back-pressure  is significantly higher after
>resuming
>> from a save point.
>>
>> Because that job makes heavy use of stateful functions with
>> RocksDBStateBackend ,
>>
>> I'm suspecting that this is the cause of performance degradation.
>>
>> Does anyone encounter simillar issues or have any tips for debugging
>?
>>
>>
>> I'm using Flink 1.3.2 with YARN in detached mode.
>>
>>
>> Regards,
>>
>> Kien
>>
>>

Mime
View raw message