Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8AA8619A0C for ; Thu, 10 Mar 2016 00:36:04 +0000 (UTC) Received: (qmail 2089 invoked by uid 500); 10 Mar 2016 00:36:04 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 2002 invoked by uid 500); 10 Mar 2016 00:36:04 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 1992 invoked by uid 99); 10 Mar 2016 00:36:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2016 00:36:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C984F1806DC for ; Thu, 10 Mar 2016 00:36:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.85 X-Spam-Level: X-Spam-Status: No, score=0.85 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.329, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 3lRV1M7bSgkT for ; Thu, 10 Mar 2016 00:36:02 +0000 (UTC) Received: from nm15-vm0.bullet.mail.ne1.yahoo.com (nm15-vm0.bullet.mail.ne1.yahoo.com [98.138.91.70]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 27AD35F1BE for ; Thu, 10 Mar 2016 00:36:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1457570154; bh=vy88uxGFJ2iEX8TChOLMYw0P1JrHT2h+2RDmo8xMDEk=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=HVxJBVfLQg5skp9ATbdTWKkGZNBHpQWJv/S7SRZneOxLI8YocMaOPwCm7rDmmamqubL2DX79An+0lrSTPfOJD2YFM47HSotZAR3ZGTFbXnLsoOabhx92CHX+nmE7hvUxfru1TeTROzb/MYeO+Hst7KmQqsIwqt2I+n/64y9PL4bqvpRnSpyAyIhi8cVl4gP6s+ZkJvQwuzSN+NyscmfBa7gFMkzuDO18YVRVStGT1B/UiJw6PVNvjQkBGJzNqnBm6lVrQCRZWrNSMPhGEZA2wYJk1v4XDvOcavc7RVcpgZb3FiogysRNHq7q9PIaPjMKEugr9NF9lYcX/LFsFL7YsQ== Received: from [98.138.226.178] by nm15.bullet.mail.ne1.yahoo.com with NNFMP; 10 Mar 2016 00:35:54 -0000 Received: from [98.138.89.192] by tm13.bullet.mail.ne1.yahoo.com with NNFMP; 10 Mar 2016 00:35:54 -0000 Received: from [127.0.0.1] by omp1050.mail.ne1.yahoo.com with NNFMP; 10 Mar 2016 00:35:54 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 546881.19041.bm@omp1050.mail.ne1.yahoo.com X-YMail-OSG: Dms5eQgVM1nIQ9Fs4tz.9tAi1eVE5RvcFC7rRqObqQeZdYLmtA14Juuwz.seoo_ OoyeOyiFN_6rK0bCMobDWgVWjY6TwD_BCY9pObj2pVsF65hF9GBNLBnH0rSD0hHGL7EDsYrNktA7 Hz0XCOI5l3ZaqDSth8ERfaaD_y1rc7EOfpNh6PAtghGWui3QwEoDbgJR4iljSaeqduSGzj5Bx9d1 WiqO11IUz7Vl8jOL_6iSxlIypGzx4tIAKJuOCEYZVbwPgggbZd.wP_tz57sL0LDg3zCOlNfJY_IQ KIohqfQICJsh.qzufRODy8nP4R.tSKrjdcc7MQQBYPRIlVJnLwSk2KpFhgBx2fBj64Bdo3b5cink 41Ihy347RDTTpbsbA00xbMS_mClB0Q7KYVJ1TSnGSoANkUvmu6A129Gh16jI4jSalOuW9TmOGBB5 hwfy1IJzTBAMwBOibyi41F1lnD_9YF4mJ3O9zAxzvoJywfgm_G5DuUwvfIFeAOY6U5n_fqQwsQU4 Yjh5SvWGG4g-- Received: by 98.138.105.211; Thu, 10 Mar 2016 00:35:54 +0000 Date: Thu, 10 Mar 2016 00:35:53 +0000 (UTC) From: Vijay Srinivasaraghavan Reply-To: Vijay Srinivasaraghavan To: Ufuk Celebi , "user@flink.apache.org" Message-ID: <2102807584.5657526.1457570153718.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: References: Subject: Re: Checkpoint MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_5657525_1217309894.1457570153714" ------=_Part_5657525_1217309894.1457570153714 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Ufuk, I have increased the sampling size to 1000 and decreased the refresh interv= al by half. In my Kafka topic I have pumped million messages which is read = by KafkaConsumer pipeline and then pass it to a transofmation step where I = have introduced sleep (3 sec) for every single message received and the fin= al step is HDFS sink using RollingSinc API. jobmanager.web.backpressure.num-samples: 1000 jobmanager.web.backpressure.refresh-interval: 30000 I was hoping to see the backpressure tab from UI to display some warning bu= t I still see "OK" message. This makes me wonder if I am testing the backpressure scenario properly or = not?=C2=A0 RegardsVijay On Monday, March 7, 2016 3:19 PM, Ufuk Celebi wrote: =20 Hey Vijay! On Mon, Mar 7, 2016 at 8:42 PM, Vijay Srinivasaraghavan wrote: > 3) How can I simulate and verify backpressure? I have introduced some del= ay > (Thread Sleep) in the job before the sink but the "backpressure" tab from= UI > does not show any indication of whether backpressure is working or not. If a task is slow, it is back pressuring upstream tasks, e.g. if your transformations have the sleep, the sources should be back pressured. It can happen that even with the sleep the tasks still produce their data as fast as they can and hence no back pressure is indicated in the web interface. You can increase the sleep to check this. The mechanism used to determine back pressure is based on sampling the stack traces of running tasks. You can increase the number of samples and/or decrease the delay between samples via config parameters shown in [1]. It can happen that the samples miss the back pressure indicators, but usually the defaults work fine. [1] https://ci.apache.org/projects/flink/flink-docs-master/setup/config.htm= l#jobmanager-web-frontend ------=_Part_5657525_1217309894.1457570153714 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Ufuk,

I have increased the sam= pling size to 1000 and decreased the refresh interval by half. In my Kafka = topic I have pumped million messages which is read by KafkaConsumer pipelin= e and then pass it to a transofmation step where I have introduced sleep (3= sec) for every single message received and the final step is HDFS sink usi= ng RollingSinc API.

jobmanager.web.backpressure.= num-samples: 1000
jobmanager.web.backpressure.refresh-interval: 30000


I was hoping to see the backpressure tab from UI to d= isplay some warning but I still see "OK" message.
<= br clear=3D"none">
This makes me wonder if I am tes= ting the backpressure scenario properly or not? 

Regards
Vijay

=
On Monday, March 7, 2016 3:19 PM, Ufuk Celebi &= lt;uce@apache.org> wrote:


Hey Vijay!

On Mon, Mar 7, 2016 at 8:42 PM, Vijay Srinivasaraghava= n
<vijikarthi@yahoo.c= om> wrote:
> 3) How can I simulate and verify b= ackpressure? I have introduced some delay
> (Thread Sl= eep) in the job before the sink but the "backpressure" tab from UI
> does not show any indication of whether backpressure is work= ing or not.

If a task is slow, it is b= ack pressuring upstream tasks, e.g. if your
transformatio= ns have the sleep, the sources should be back pressured.
= It can happen that even with the sleep the tasks still produce their
data as fast as they can and hence no back pressure is indicate= d in
the web interface. You can increase the sleep to che= ck this.

The mechanism used to determi= ne back pressure is based on sampling the
stack traces of= running tasks. You can increase the number of samples
an= d/or decrease the delay between samples via config parameters shown
in [1]. It can happen that the samples miss the back pressureindicators, but usually the defaults work fine.


[1]
https://ci.apache.org/projects= /flink/flink-docs-master/setup/config.html#jobmanager-web-frontend



------=_Part_5657525_1217309894.1457570153714--