storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <roshan_n...@yahoo.com>
Subject Re: When is the MAX_SPOUT_PENDING limit applied
Date Wed, 03 Apr 2019 21:12:25 GMT
It’s kind of implied from prior responses but just wanted to emphasize  .. it is not used
if ACKing is disabled (I.e acker count = 0)


Sent from Yahoo Mail for iPhone


On Wednesday, April 3, 2019, 11:50 AM, Stig Rohde Døssing <stigdoessing@gmail.com>
wrote:

Supplementing the other great answers here, Storm checks max spout pending before it invokes
nextTuple on the spout. The spout executor calls nextTuple in a loop. If the spout task has
more pending tuples (i.e. emitted but not acked or failed or timed out) than max spout pending,
the executor won't call nextTuple, but instead wait for a bit before it checks again. The
relevant code is here https://github.com/apache/storm/blob/9680928a0c6e26a8a2d3e5ceb9a632579546353d/storm-client/src/jvm/org/apache/storm/executor/spout/SpoutExecutor.java#L190

Den ons. 3. apr. 2019 kl. 15.00 skrev Alessio Pagliari <pagliari@i3s.unice.fr>:

Hi Jayant,
The MAX_SPOUT_PENDING is applied in each of the spouts. 
Each spout take count of all the tuples it sends, increasing it for each emit and decreasing
it when it receive the final ack message for a “pending tuple” (= a tuple that is sent
but it’s still waiting an ack). If the number of pending tuples raises above the value
you set, let’s say 500, the spouts will automatically slow down, until they reach an emission
rate such that the number of pending tuples stabilise under 500.This mechanism is applied
per spout, meaning that each spout can be limited in different times, and the MAX_SPOUT_PENDING
represents the maximum number of non-acked tuples that a single spout will allow.
Alessio

On 3 Apr 2019, at 14:45, Joshua Martell <joshua.martell@gmail.com> wrote:
I’m pretty sure it’s per spout task. And they’re not coordinated. Each task gets its
own count. 

Joshua
On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma <sharmajayant27@gmail.com> wrote:

Hi,

Can someone please explain at what point storm checks and applies MAX_SPOUT_PENDING limit.
Also, is this limit applied per executor(or Task if that's the case) of spout or aggregated
and applied over all the executors of spout.Suppose I have 3 executors and 3 tasks of spouts,
each of them fetch 1 message from input source. If my MAX_SPOUT_PENDING is 2, will only 2
executors be able to send the tuple forward or all 3 will send and further nextTuple calls
on all will be blocked?
Thanks,Jayant Sharma







Mime
View raw message