Return-Path: X-Original-To: apmail-apex-users-archive@minotaur.apache.org Delivered-To: apmail-apex-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2470D18221 for ; Mon, 1 Feb 2016 07:17:10 +0000 (UTC) Received: (qmail 77182 invoked by uid 500); 1 Feb 2016 07:16:11 -0000 Delivered-To: apmail-apex-users-archive@apex.apache.org Received: (qmail 77128 invoked by uid 500); 1 Feb 2016 07:16:11 -0000 Mailing-List: contact users-help@apex.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@apex.incubator.apache.org Delivered-To: mailing list users@apex.incubator.apache.org Received: (qmail 77118 invoked by uid 99); 1 Feb 2016 07:16:11 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2016 07:16:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id EEEB8C08FF for ; Mon, 1 Feb 2016 07:16:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.999 X-Spam-Level: ** X-Spam-Status: No, score=2.999 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=datatorrent-com.20150623.gappssmtp.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id O7KQ4LJ8rjtm for ; Mon, 1 Feb 2016 07:16:05 +0000 (UTC) Received: from mail-qg0-f52.google.com (mail-qg0-f52.google.com [209.85.192.52]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id EA4F820271 for ; Mon, 1 Feb 2016 07:16:04 +0000 (UTC) Received: by mail-qg0-f52.google.com with SMTP id e32so111643565qgf.3 for ; Sun, 31 Jan 2016 23:16:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datatorrent-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=Oy0tTlrCG5uvHLUFLljuW7JnfaI1COVvb1IidzaluAA=; b=iEKRkVv9csZ3cCP9fXv0tLQ75H8H6TH5/mopKbnZ8YmfM1EuhhhAs+Dlem4+JpXGPn 0xDkYbCAMGoElVtnj5Z/zUbR5mHZPWL+Wq6izmyMriSLvdblJePQlN0RgR68/Rk/5RL1 vN8jRormK8WnPjEqJRbgSqr07tXumkQrzaGWE3JMCC2aEIZsFoCGKPWuIM8jI8r/ZKli 16Q9SxpLV+brnIm7W7RwZLuhe2jl3EzwiYZUKAX2awlFGfVX6+IN4M7zpw79XtThCeWK Vvjrg7u1IuGUErp/X6RIwwE4o+UF5K5iJIhtO55aBoDqUv66F6lx8aKQgetmKVuCC7Ll TlKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=Oy0tTlrCG5uvHLUFLljuW7JnfaI1COVvb1IidzaluAA=; b=hYYt30pK9rh1bRwIEcjNQbDXMOvYSvW2hf1oseZEQ3EZH/TKriSaX7y2IBp/jnIECi 4d+UE+0+niT3xMC9RmqGh8T+u24t6YqFICVmaKWuUhdkD5Wf/G55QFq/HaOb38XF/VFm EqFOAJ9CPvzh7uH/I4nyF0WrSYohXoMrm2v/rEe/pwrGwDSLAASv2fWYoco0odQ9LnSM ztFtmxg2Od7QhTPL/8GIG0luWxijI7Oe8IhrUdVv/HasqMoHGiQ4mqLDZafWiBOezB2H TVkRd7k+56zM/ar742SucWiqFd3FbIhrjTqXAQ9BYOgy3AI9/wkMSxYOZMuePB7XuKe0 0t5A== X-Gm-Message-State: AG10YOT+w2zslPMqgbIvFPvW12I1mJVebCIU38pRVM2heGHYO7aoBhC4tzx3tqfpJFZOJCGuKdVl4SS+l3Yje2Lg X-Received: by 10.140.162.2 with SMTP id i2mr25098935qhi.44.1454310958062; Sun, 31 Jan 2016 23:15:58 -0800 (PST) MIME-Version: 1.0 Received: by 10.140.18.227 with HTTP; Sun, 31 Jan 2016 23:15:28 -0800 (PST) In-Reply-To: References: <56AED9EB.7040706@datatorrent.com> From: Pramod Immaneni Date: Sun, 31 Jan 2016 23:15:28 -0800 Message-ID: Subject: Re: What is the backpressure story ? To: users@apex.incubator.apache.org Content-Type: multipart/alternative; boundary=001a113a4cbc0c73bb052ab0282a --001a113a4cbc0c73bb052ab0282a Content-Type: text/plain; charset=UTF-8 For natural back pressure to work in all cases the memory issues need to be addressed and work is being done on that. That was my point in the earlier email. Thanks On Sun, Jan 31, 2016 at 9:54 PM, Chinmay Kolhatkar wrote: > When an upstream operator pushes data faster than the buffer server can > spool it to disk, the buffer server disables reads from the upstream > operator putting a back pressure on the upstream operator once the limit is > reached > > [CK] If I read it correctly, this means that if the point when back > pressure takes effect on upstream operators is dependent on disk > performance. If that is true, would it make sense to make this independent > of disk performance? > > > > On Mon, Feb 1, 2016 at 9:37 AM, Vlad Rozov > wrote: > >> Thomas is correct. When buffer server spooling is enabled (default >> behavior since 3.0), the buffer server limits its memory usage and starts >> spooling to disk once half of the specified limit is reached. When an >> upstream operator pushes data faster than the buffer server can spool it to >> disk, the buffer server disables reads from the upstream operator putting a >> back pressure on the upstream operator once the limit is reached. It gives >> ability for a downstream operator(s) to catch up with the data already >> pushed to the buffer server reducing amount of memory the buffer server >> uses. Once it drops below the limit, the buffer server enables reads from >> the upstream operator. By default buffer server is allowed to use 512 MB, >> and it can be configured using BUFFER_MEMORY_MB port attribute. >> >> When spooling is disabled (using another port attribute BUFFER_SPOOLING), >> the buffer server does not limit its memory usage, so if it is a temporary >> slowdown in a down stream operator and there is sufficient amount of >> memory, the buffer server will not crash. Only when downstream operator(s) >> are not capable to keep up with the upstream operator, the buffer server >> and JVM may run out of allocated memory. >> >> There are several JIRAs open to enhance the buffer server to enable back >> pressure mechanism when spooling is disabled and also limit amount of disk >> storage that the buffer server may use for spooling. >> >> Vlad >> >> >> On 1/31/16 16:24, Thomas Weise wrote: >> >> That's incorrect. Backpressure works when spooling is enabled (which is >> default). It's not handled only when you turn spooling off explicitly. >> >> On Sun, Jan 31, 2016 at 3:50 PM, Sandesh Hegde >> wrote: >> >>> According to Vlad, disabling the spooling will crash the buffer server >>> after it runs out of memory. >>> >>> It means Apex doesn't have a mechanism to handle backpressure yet. >>> >>> On Fri, Jan 29, 2016 at 9:34 AM Pramod Immaneni >>> wrote: >>> >>>> By default buffer spooling is enabled so data gets spooled to file >>>> system once the buffer limits are reached, there will be some slow down but >>>> upstream will continue to process, if buffer spooling is disabled then when >>>> the buffers are filled the sender is blocked and this back pressure will >>>> propagate upstream to the first operator. >>>> >>>> On Fri, Jan 29, 2016 at 7:47 AM, Sandesh Hegde < >>>> sandesh@datatorrent.com> wrote: >>>> >>>>> Hello Team, >>>>> >>>>> My understanding of the backpressure in Apex is, Buffer server will >>>>> slow down ( because of TCP/IP congestion control ) the upstream operator if >>>>> the downstream is slow. Is there more to it? >>>>> I don't see this topic covered in docs. >>>>> >>>>> Thanks >>>>> Sandesh >>>>> >>>>> >>>>> >>>>> >>>> >> >> > --001a113a4cbc0c73bb052ab0282a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
For natural back pressure to work in all cases the memory = issues need to be addressed and work is being done on that. That was my poi= nt in the earlier email.

Thanks

On Sun, Jan 31, 2016 at 9:54 P= M, Chinmay Kolhatkar <chinmay@datatorrent.com> wrote:<= br>
When an upstream operator pushes data faster than= the buffer server can spool it to disk, the buffer server disables reads f= rom the upstream operator putting a back pressure on the upstream operator = once the limit is reached
<= br>
[CK] If I read= it correctly, this means that if the point when back pressure takes effect= on upstream operators is dependent on disk performance. If that is true, w= ould it make sense to make this independent of disk performance?



On Mon,= Feb 1, 2016 at 9:37 AM, Vlad Rozov <v.rozov@datatorrent.com>= wrote:
=20 =20 =20
Thomas is correct. When buffer server spooling is enabled (default behavior since 3.0), the buffer server limits its memory usage and starts spooling to disk once half of the specified limit is reached. When an upstream operator pushes data faster than the buffer server can spool it to disk, the buffer server disables reads from the upstream operator putting a back pressure on the upstream operator once the limit is reached. It gives ability for a downstream operator(s) to catch up with the data already pushed to the buffer server reducing amount of memory the buffer server uses. Once it drops below the limit, the buffer server enables reads from the upstream operator. By default buffer server is allowed to use 512 MB, and it can be configured using BUFFER_MEMORY_MB =20 port attribute.

When spooling is disabled (using another port attribute BUFFER_SPOOLING), the buffer server does not limit its memory usage, so if it is a temporary slowdown in a down stream operator and there is sufficient amount of memory, the buffer server will not crash. Only when downstream operator(s) are not capable to keep up with the upstream operator, the buffer server and JVM may run out of allocated memory.

There are several JIRAs open to enhance the buffer server to enable back pressure mechanism when spooling is disabled and also limit amount of disk storage that the buffer server may use for spooling.

Vlad


On 1/31/16 16:24, Thomas Weise wrote:
That's incorrect. Backpressure works when spooli= ng is enabled (which is default). It's not handled only when you turn spooling off=C2=A0explicitly.

On Sun, Jan 31, 2016 at 3:50 PM, Sandesh Hegde <sandesh@datatorrent.com> wrote:
According to Vlad, disabling the spooling will crash the buffer server after it runs out of memory.

It means Apex doesn't have a mechanism to handle backpressure yet.=C2=A0

On Fri, Jan 29, 2016 at 9:34 AM Pramod Immaneni <pramod@datatorrent.com> wrote:
By default buffer spooling is enabled so data gets spooled to file system once the buffer limits are reached, there will be some slow down but upstream will continue to process, if buffer spooling is disabled then when the buffers are filled the sender is blocked and this back pressure will propagate upstream to the first operator.

On Fri, Jan 29, 2016 at 7:47 AM, Sandesh Hegde <sandesh@datatorrent.com>= ; wrote:
Hello Team,

My understanding of the backpressure in Apex is, Buffer server will slow down ( because of TCP/IP congestion control ) the upstream operator if the downstream is slow. Is there more to it?=C2=A0
I don't see this topic covered in docs= .

Thanks
Sandesh=C2=A0








--001a113a4cbc0c73bb052ab0282a--