apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vlad Rozov <v.ro...@datatorrent.com>
Subject Re: What is the backpressure story ?
Date Mon, 01 Feb 2016 15:45:04 GMT
IMO, the further discussion should be continued on the dev@apex list.

As I mention, there are several enhancement requests open for the buffer 
server related to the back pressure behavior and limiting memory usage 
when spooling is disabled is only one of them. It is also necessary to 
limit amount of disk usage when spooling is enabled. Those are the major 
ones. There is another JIRA for changing how the buffer server reads 
blocks spooled to disk. I plan to work on those requests once the 
network prototype is ready, but possibly somebody from the Apex 
community wants to take a look.

The netlet memory usage is separate from the buffer server memory usage. 
The buffer server has it's own pool of memory blocks separate from the 
netlet queue. The netlet queue is already upper bounded, but the problem 
is that it limits number of byte arrays that may be waiting in the 
queue, not the total memory consumption. There is pull request in 
progress to limit netlet memory usage. It is still necessary to look 
into the buffer server JIRAs.

Vlad

On 1/31/16 23:15, Pramod Immaneni wrote:
> For natural back pressure to work in all cases the memory issues need 
> to be addressed and work is being done on that. That was my point in 
> the earlier email.
>
> Thanks
>
> On Sun, Jan 31, 2016 at 9:54 PM, Chinmay Kolhatkar 
> <chinmay@datatorrent.com <mailto:chinmay@datatorrent.com>> wrote:
>
>     When an upstream operator pushes data faster than the buffer
>     server can spool it to disk, the buffer server disables reads from
>     the upstream operator putting a back pressure on the upstream
>     operator once the limit is reached
>
>     [CK] If I read it correctly, this means that if the point when
>     back pressure takes effect on upstream operators is dependent on
>     disk performance. If that is true, would it make sense to make
>     this independent of disk performance?
>
>
>
>     On Mon, Feb 1, 2016 at 9:37 AM, Vlad Rozov
>     <v.rozov@datatorrent.com <mailto:v.rozov@datatorrent.com>> wrote:
>
>         Thomas is correct. When buffer server spooling is enabled
>         (default behavior since 3.0), the buffer server limits its
>         memory usage and starts spooling to disk once half of the
>         specified limit is reached. When an upstream operator pushes
>         data faster than the buffer server can spool it to disk, the
>         buffer server disables reads from the upstream operator
>         putting a back pressure on the upstream operator once the
>         limit is reached. It gives ability for a downstream
>         operator(s) to catch up with the data already pushed to the
>         buffer server reducing amount of memory the buffer server
>         uses. Once it drops below the limit, the buffer server enables
>         reads from the upstream operator. By default buffer server is
>         allowed to use 512 MB, and it can be configured using
>         BUFFER_MEMORY_MB port attribute.
>
>         When spooling is disabled (using another port attribute
>         BUFFER_SPOOLING), the buffer server does not limit its memory
>         usage, so if it is a temporary slowdown in a down stream
>         operator and there is sufficient amount of memory, the buffer
>         server will not crash. Only when downstream operator(s) are
>         not capable to keep up with the upstream operator, the buffer
>         server and JVM may run out of allocated memory.
>
>         There are several JIRAs open to enhance the buffer server to
>         enable back pressure mechanism when spooling is disabled and
>         also limit amount of disk storage that the buffer server may
>         use for spooling.
>
>         Vlad
>
>
>         On 1/31/16 16:24, Thomas Weise wrote:
>>         That's incorrect. Backpressure works when spooling is enabled
>>         (which is default). It's not handled only when you turn
>>         spooling off explicitly.
>>
>>         On Sun, Jan 31, 2016 at 3:50 PM, Sandesh Hegde
>>         <sandesh@datatorrent.com <mailto:sandesh@datatorrent.com>> wrote:
>>
>>             According to Vlad, disabling the spooling will crash the
>>             buffer server after it runs out of memory.
>>
>>             It means Apex doesn't have a mechanism to handle
>>             backpressure yet.
>>
>>             On Fri, Jan 29, 2016 at 9:34 AM Pramod Immaneni
>>             <pramod@datatorrent.com <mailto:pramod@datatorrent.com>>
>>             wrote:
>>
>>                 By default buffer spooling is enabled so data gets
>>                 spooled to file system once the buffer limits are
>>                 reached, there will be some slow down but upstream
>>                 will continue to process, if buffer spooling is
>>                 disabled then when the buffers are filled the sender
>>                 is blocked and this back pressure will propagate
>>                 upstream to the first operator.
>>
>>                 On Fri, Jan 29, 2016 at 7:47 AM, Sandesh Hegde
>>                 <sandesh@datatorrent.com
>>                 <mailto:sandesh@datatorrent.com>> wrote:
>>
>>                     Hello Team,
>>
>>                     My understanding of the backpressure in Apex is,
>>                     Buffer server will slow down ( because of TCP/IP
>>                     congestion control ) the upstream operator if the
>>                     downstream is slow. Is there more to it?
>>                     I don't see this topic covered in docs.
>>
>>                     Thanks
>>                     Sandesh
>>
>>
>>
>>
>>
>
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message