storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessio Pagliari <pagli...@i3s.unice.fr>
Subject Re: When is the MAX_SPOUT_PENDING limit applied
Date Wed, 03 Apr 2019 13:00:19 GMT
Hi Jayant,

The MAX_SPOUT_PENDING is applied in each of the spouts. 

Each spout take count of all the tuples it sends, increasing it for each emit and decreasing
it when it receive the final ack message for a “pending tuple” (= a tuple that is sent
but it’s still waiting an ack). 
If the number of pending tuples raises above the value you set, let’s say 500, the spouts
will automatically slow down, until they reach an emission rate such that the number of pending
tuples stabilise under 500.
This mechanism is applied per spout, meaning that each spout can be limited in different times,
and the MAX_SPOUT_PENDING represents the maximum number of non-acked tuples that a single
spout will allow.

Alessio

> On 3 Apr 2019, at 14:45, Joshua Martell <joshua.martell@gmail.com> wrote:
> 
> I’m pretty sure it’s per spout task. And they’re not coordinated. Each task gets
its own count. 
> 
> Joshua
> On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma <sharmajayant27@gmail.com <mailto:sharmajayant27@gmail.com>>
wrote:
> Hi,
> 
> Can someone please explain at what point storm checks and applies MAX_SPOUT_PENDING limit.
Also, is this limit applied per executor(or Task if that's the case) of spout or aggregated
and applied over all the executors of spout.
> Suppose I have 3 executors and 3 tasks of spouts, each of them fetch 1 message from input
source. If my MAX_SPOUT_PENDING is 2, will only 2 executors be able to send the tuple forward
or all 3 will send and further nextTuple calls on all will be blocked?
> 
> Thanks,
> Jayant Sharma


Mime
View raw message