storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stig Rohde Døssing <>
Subject Re: When is the MAX_SPOUT_PENDING limit applied
Date Wed, 03 Apr 2019 18:50:48 GMT
Supplementing the other great answers here, Storm checks max spout pending
before it invokes nextTuple on the spout. The spout executor calls
nextTuple in a loop. If the spout task has more pending tuples (i.e.
emitted but not acked or failed or timed out) than max spout pending, the
executor won't call nextTuple, but instead wait for a bit before it checks
again. The relevant code is here

Den ons. 3. apr. 2019 kl. 15.00 skrev Alessio Pagliari <>:

> Hi Jayant,
> The MAX_SPOUT_PENDING is applied in each of the spouts.
> Each spout take count of all the tuples it sends, increasing it for each
> emit and decreasing it when it receive the final ack message for a “pending
> tuple” (= a tuple that is sent but it’s still waiting an ack).
> If the number of pending tuples raises above the value you set, let’s say
> 500, the spouts will automatically slow down, until they reach an emission
> rate such that the number of pending tuples stabilise under 500.
> This mechanism is applied per spout, meaning that each spout can be
> limited in different times, and the MAX_SPOUT_PENDING represents the
> maximum number of non-acked tuples that a single spout will allow.
> Alessio
> On 3 Apr 2019, at 14:45, Joshua Martell <> wrote:
> I’m pretty sure it’s per spout task. And they’re not coordinated. Each
> task gets its own count.
> Joshua
> On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma <>
> wrote:
>> Hi,
>> Can someone please explain at what point storm checks and applies
>> MAX_SPOUT_PENDING limit. Also, is this limit applied per executor(or Task
>> if that's the case) of spout or aggregated and applied over all the
>> executors of spout.
>> Suppose I have 3 executors and 3 tasks of spouts, each of them fetch 1
>> message from input source. If my MAX_SPOUT_PENDING is 2, will only 2
>> executors be able to send the tuple forward or all 3 will send and further
>> nextTuple calls on all will be blocked?
>> Thanks,
>> Jayant Sharma

View raw message