flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay Srinivasaraghavan <vijikar...@yahoo.com>
Subject Re: Checkpoint
Date Thu, 10 Mar 2016 00:35:53 GMT
Hi Ufuk,
I have increased the sampling size to 1000 and decreased the refresh interval by half. In
my Kafka topic I have pumped million messages which is read by KafkaConsumer pipeline and
then pass it to a transofmation step where I have introduced sleep (3 sec) for every single
message received and the final step is HDFS sink using RollingSinc API.
jobmanager.web.backpressure.num-samples: 1000
jobmanager.web.backpressure.refresh-interval: 30000

I was hoping to see the backpressure tab from UI to display some warning but I still see "OK"
This makes me wonder if I am testing the backpressure scenario properly or not? 
    On Monday, March 7, 2016 3:19 PM, Ufuk Celebi <uce@apache.org> wrote:

 Hey Vijay!

On Mon, Mar 7, 2016 at 8:42 PM, Vijay Srinivasaraghavan
<vijikarthi@yahoo.com> wrote:
> 3) How can I simulate and verify backpressure? I have introduced some delay
> (Thread Sleep) in the job before the sink but the "backpressure" tab from UI
> does not show any indication of whether backpressure is working or not.

If a task is slow, it is back pressuring upstream tasks, e.g. if your
transformations have the sleep, the sources should be back pressured.
It can happen that even with the sleep the tasks still produce their
data as fast as they can and hence no back pressure is indicated in
the web interface. You can increase the sleep to check this.

The mechanism used to determine back pressure is based on sampling the
stack traces of running tasks. You can increase the number of samples
and/or decrease the delay between samples via config parameters shown
in [1]. It can happen that the samples miss the back pressure
indicators, but usually the defaults work fine.

[1] https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#jobmanager-web-frontend

View raw message