From commits-return-43056-archive-asf-public=cust-asf.ponee.io@pulsar.apache.org Mon Nov 25 22:39:52 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id EDDDD180607 for ; Mon, 25 Nov 2019 23:39:51 +0100 (CET) Received: (qmail 78075 invoked by uid 500); 25 Nov 2019 22:39:51 -0000 Mailing-List: contact commits-help@pulsar.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@pulsar.apache.org Delivered-To: mailing list commits@pulsar.apache.org Received: (qmail 78066 invoked by uid 99); 25 Nov 2019 22:39:51 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Nov 2019 22:39:51 +0000 From: GitBox To: commits@pulsar.apache.org Subject: [GitHub] [pulsar] rdhabalia commented on issue #5513: Publish rate limit on broker to avoid OOM Message-ID: <157472159122.12737.8674011862840424577.gitbox@gitbox.apache.org> Date: Mon, 25 Nov 2019 22:39:51 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit rdhabalia commented on issue #5513: Publish rate limit on broker to avoid OOM URL: https://github.com/apache/pulsar/issues/5513#issuecomment-558372993 >> @merlimat There's already a limit per-connection of max outstanding entries between broker and bookies. Do you have a precise scenario for this to happen? >> @sijie create a topic with 20 partitions. with parallelism == 20. The producer setting in the pulsar sink is the default producer setting. I gave a try with similar setup and somehow I couldn't reproduce it. and reason broker doesn't go OOM because I think what @merlimat mentioned. Broker restricts max pending publish request per connection. https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java#L130 also, introducing counter across all topics for throttling can cause bottleneck while publishing for all topics so, this feature might not be recommended for most of the users. so, I would recommend to depend on maxPendingRequestPerConnection rather adding more complexity and if that's not working then it's worth to investigate why it disabling channel still cause OOM. I have also created #5742 which can allow users to configure max-pending requests per connection if needed. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org With regards, Apache Git Services