apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: What is the backpressure story ?
Date Mon, 01 Feb 2016 07:15:28 GMT
For natural back pressure to work in all cases the memory issues need to be
addressed and work is being done on that. That was my point in the earlier
email.

Thanks

On Sun, Jan 31, 2016 at 9:54 PM, Chinmay Kolhatkar <chinmay@datatorrent.com>
wrote:

> When an upstream operator pushes data faster than the buffer server can
> spool it to disk, the buffer server disables reads from the upstream
> operator putting a back pressure on the upstream operator once the limit is
> reached
>
> [CK] If I read it correctly, this means that if the point when back
> pressure takes effect on upstream operators is dependent on disk
> performance. If that is true, would it make sense to make this independent
> of disk performance?
>
>
>
> On Mon, Feb 1, 2016 at 9:37 AM, Vlad Rozov <v.rozov@datatorrent.com>
> wrote:
>
>> Thomas is correct. When buffer server spooling is enabled (default
>> behavior since 3.0), the buffer server limits its memory usage and starts
>> spooling to disk once half of the specified limit is reached. When an
>> upstream operator pushes data faster than the buffer server can spool it to
>> disk, the buffer server disables reads from the upstream operator putting a
>> back pressure on the upstream operator once the limit is reached. It gives
>> ability for a downstream operator(s) to catch up with the data already
>> pushed to the buffer server reducing amount of memory the buffer server
>> uses. Once it drops below the limit, the buffer server enables reads from
>> the upstream operator. By default buffer server is allowed to use 512 MB,
>> and it can be configured using BUFFER_MEMORY_MB port attribute.
>>
>> When spooling is disabled (using another port attribute BUFFER_SPOOLING),
>> the buffer server does not limit its memory usage, so if it is a temporary
>> slowdown in a down stream operator and there is sufficient amount of
>> memory, the buffer server will not crash. Only when downstream operator(s)
>> are not capable to keep up with the upstream operator, the buffer server
>> and JVM may run out of allocated memory.
>>
>> There are several JIRAs open to enhance the buffer server to enable back
>> pressure mechanism when spooling is disabled and also limit amount of disk
>> storage that the buffer server may use for spooling.
>>
>> Vlad
>>
>>
>> On 1/31/16 16:24, Thomas Weise wrote:
>>
>> That's incorrect. Backpressure works when spooling is enabled (which is
>> default). It's not handled only when you turn spooling off explicitly.
>>
>> On Sun, Jan 31, 2016 at 3:50 PM, Sandesh Hegde <sandesh@datatorrent.com>
>> wrote:
>>
>>> According to Vlad, disabling the spooling will crash the buffer server
>>> after it runs out of memory.
>>>
>>> It means Apex doesn't have a mechanism to handle backpressure yet.
>>>
>>> On Fri, Jan 29, 2016 at 9:34 AM Pramod Immaneni <pramod@datatorrent.com>
>>> wrote:
>>>
>>>> By default buffer spooling is enabled so data gets spooled to file
>>>> system once the buffer limits are reached, there will be some slow down but
>>>> upstream will continue to process, if buffer spooling is disabled then when
>>>> the buffers are filled the sender is blocked and this back pressure will
>>>> propagate upstream to the first operator.
>>>>
>>>> On Fri, Jan 29, 2016 at 7:47 AM, Sandesh Hegde <
>>>> <sandesh@datatorrent.com>sandesh@datatorrent.com> wrote:
>>>>
>>>>> Hello Team,
>>>>>
>>>>> My understanding of the backpressure in Apex is, Buffer server will
>>>>> slow down ( because of TCP/IP congestion control ) the upstream operator
if
>>>>> the downstream is slow. Is there more to it?
>>>>> I don't see this topic covered in docs.
>>>>>
>>>>> Thanks
>>>>> Sandesh
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
>

Mime
View raw message