storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stig Rohde Døssing <stigdoess...@gmail.com>
Subject Re: When is the MAX_SPOUT_PENDING limit applied
Date Wed, 03 Apr 2019 18:50:48 GMT
Supplementing the other great answers here, Storm checks max spout pending
before it invokes nextTuple on the spout. The spout executor calls
nextTuple in a loop. If the spout task has more pending tuples (i.e.
emitted but not acked or failed or timed out) than max spout pending, the
executor won't call nextTuple, but instead wait for a bit before it checks
again. The relevant code is here
https://github.com/apache/storm/blob/9680928a0c6e26a8a2d3e5ceb9a632579546353d/storm-client/src/jvm/org/apache/storm/executor/spout/SpoutExecutor.java#L190

Den ons. 3. apr. 2019 kl. 15.00 skrev Alessio Pagliari <
pagliari@i3s.unice.fr>:

> Hi Jayant,
>
> The MAX_SPOUT_PENDING is applied in each of the spouts.
>
> Each spout take count of all the tuples it sends, increasing it for each
> emit and decreasing it when it receive the final ack message for a “pending
> tuple” (= a tuple that is sent but it’s still waiting an ack).
> If the number of pending tuples raises above the value you set, let’s say
> 500, the spouts will automatically slow down, until they reach an emission
> rate such that the number of pending tuples stabilise under 500.
> This mechanism is applied per spout, meaning that each spout can be
> limited in different times, and the MAX_SPOUT_PENDING represents the
> maximum number of non-acked tuples that a single spout will allow.
>
> Alessio
>
> On 3 Apr 2019, at 14:45, Joshua Martell <joshua.martell@gmail.com> wrote:
>
> I’m pretty sure it’s per spout task. And they’re not coordinated. Each
> task gets its own count.
>
> Joshua
> On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma <sharmajayant27@gmail.com>
> wrote:
>
>> Hi,
>>
>> Can someone please explain at what point storm checks and applies
>> MAX_SPOUT_PENDING limit. Also, is this limit applied per executor(or Task
>> if that's the case) of spout or aggregated and applied over all the
>> executors of spout.
>> Suppose I have 3 executors and 3 tasks of spouts, each of them fetch 1
>> message from input source. If my MAX_SPOUT_PENDING is 2, will only 2
>> executors be able to send the tuple forward or all 3 will send and further
>> nextTuple calls on all will be blocked?
>>
>> Thanks,
>> Jayant Sharma
>>
>
>

Mime
View raw message