flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijiang(wangzhijiang999)" <wangzhijiang...@aliyun.com>
Subject 回复:Handling back pressure in Flink.
Date Fri, 06 Jul 2018 03:44:56 GMT
Hi Mich,

From flink-1.5.0 the network flow control is improved by credit-based mechanism whichs handles
backpressure better than before. The producer sends data based on the number of available
buffers(credit) onconsumer side. If processing time on consumer side is slower than producing
time on producer side, the data will be cached on outqueue and inqueue memories of both side
which may trigger back pressure the producer side. You can increase the number of credits
on consumer side to relieve back pressure. Eventhough the back pressure happens, the application
is still stable (will not cause OOM). I think you should not worry about that. Normally it
is better to consider TPS of both sides and set the proper paralellism to avoid back pressure
to some extent.

Zhijiang
------------------------------------------------------------------
发件人:Mich Talebzadeh <mich.talebzadeh@gmail.com>
发送时间:2018年7月4日(星期三) 20:40
收件人:user <user@flink.apache.org>
主 题:Handling back pressure in Flink.

Hi,

In spark one can handle back pressure by setting the spark conf parameter:

sparkConf.set("spark.streaming.backpressure.enabled","true")

With backpressure you make Spark Streaming application stable, i.e. receives data only as
fast as it can process it. In general one needs to ensure that your microbatching processing
time is less that your batch interval, i.e the rate that your producer sends data into Kafka.
For example this is shown in Spark GUI below for batch interval = 2 seconds



Is there such procedure in Flink please?

Thanks


Dr Mich Talebzadeh

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

http://talebzadehmich.wordpress.com

Disclaimer: Use it at your own risk.Any and all responsibility for any loss, damage or destruction
of data or any other property which may arise from relying on this email's technical content
is explicitly disclaimed. The author will in no case be liable for any monetary damages arising
from such loss, damage or destruction. 


Mime
View raw message