From user-return-14353-archive-asf-public=cust-asf.ponee.io@storm.apache.org Wed Apr 3 13:00:36 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id B474D180600 for ; Wed, 3 Apr 2019 15:00:35 +0200 (CEST) Received: (qmail 32792 invoked by uid 500); 3 Apr 2019 13:00:29 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 32782 invoked by uid 99); 3 Apr 2019 13:00:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Apr 2019 13:00:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 08532CCDCB for ; Wed, 3 Apr 2019 13:00:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3 X-Spam-Level: *** X-Spam-Status: No, score=3 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id XnVMbaH_tbE6 for ; Wed, 3 Apr 2019 13:00:27 +0000 (UTC) Received: from ip-nice01.unice.fr (ip-nice01.unice.fr [134.59.1.214]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 387CE61263 for ; Wed, 3 Apr 2019 13:00:27 +0000 (UTC) X-IronPort-AV: E=Sophos;i="5.56,253,1539640800"; d="scan'208,217";a="317885983" Received: from naxos2.unice.fr ([134.59.1.112]) by ip-nice08.unice.fr with ESMTP; 03 Apr 2019 15:00:20 +0200 Received: from walter-eco.i3s.unice.fr (walter-eco.i3s.unice.fr [134.59.133.102]) (authenticated bits=0) by naxos2.unice.fr (8.14.7/8.14.7) with ESMTP id x33Cxirs182374 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 3 Apr 2019 15:00:20 +0200 From: Alessio Pagliari Content-Type: multipart/alternative; boundary="Apple-Mail=_6722CDDB-6A99-4406-86EA-A334D0051909" Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: When is the MAX_SPOUT_PENDING limit applied Date: Wed, 3 Apr 2019 15:00:19 +0200 References: To: user@storm.apache.org In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3445.102.3) --Apple-Mail=_6722CDDB-6A99-4406-86EA-A334D0051909 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Jayant, The MAX_SPOUT_PENDING is applied in each of the spouts.=20 Each spout take count of all the tuples it sends, increasing it for each = emit and decreasing it when it receive the final ack message for a = =E2=80=9Cpending tuple=E2=80=9D (=3D a tuple that is sent but it=E2=80=99s= still waiting an ack).=20 If the number of pending tuples raises above the value you set, let=E2=80=99= s say 500, the spouts will automatically slow down, until they reach an = emission rate such that the number of pending tuples stabilise under = 500. This mechanism is applied per spout, meaning that each spout can be = limited in different times, and the MAX_SPOUT_PENDING represents the = maximum number of non-acked tuples that a single spout will allow. Alessio > On 3 Apr 2019, at 14:45, Joshua Martell = wrote: >=20 > I=E2=80=99m pretty sure it=E2=80=99s per spout task. And they=E2=80=99re= not coordinated. Each task gets its own count.=20 >=20 > Joshua > On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma > wrote: > Hi, >=20 > Can someone please explain at what point storm checks and applies = MAX_SPOUT_PENDING limit. Also, is this limit applied per executor(or = Task if that's the case) of spout or aggregated and applied over all the = executors of spout. > Suppose I have 3 executors and 3 tasks of spouts, each of them fetch 1 = message from input source. If my MAX_SPOUT_PENDING is 2, will only 2 = executors be able to send the tuple forward or all 3 will send and = further nextTuple calls on all will be blocked? >=20 > Thanks, > Jayant Sharma --Apple-Mail=_6722CDDB-6A99-4406-86EA-A334D0051909 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Jayant,

The = MAX_SPOUT_PENDING is applied in each of the spouts. 

Each spout take count of = all the tuples it sends, increasing it for each emit and decreasing it = when it receive the final ack message for a =E2=80=9Cpending tuple=E2=80=9D= (=3D a tuple that is sent but it=E2=80=99s still waiting an = ack). 
If the number of pending tuples raises = above the value you set, let=E2=80=99s say 500, the spouts will = automatically slow down, until they reach an emission rate such that the = number of pending tuples stabilise under 500.
This = mechanism is applied per spout, meaning that each spout can be limited = in different times, and the MAX_SPOUT_PENDING represents the maximum = number of non-acked tuples that a single spout will allow.

Alessio

On 3 Apr = 2019, at 14:45, Joshua Martell <joshua.martell@gmail.com> wrote:

I=E2=80=99m pretty = sure it=E2=80=99s per spout task. And they=E2=80=99re not coordinated. = Each task gets its own count.

Joshua
On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma <sharmajayant27@gmail.com> wrote:
Hi,

Can someone please explain at what point storm checks and = applies MAX_SPOUT_PENDING limit. Also, is this limit applied per = executor(or Task if that's the case) of spout or aggregated and applied = over all the executors of spout.
Suppose I have 3 = executors and 3 tasks of spouts, each of them fetch 1 message from input = source. If my MAX_SPOUT_PENDING is 2, will only 2 executors be able to = send the tuple forward or all 3 will send and further nextTuple calls on = all will be blocked?

Thanks,
Jayant Sharma

= --Apple-Mail=_6722CDDB-6A99-4406-86EA-A334D0051909--