pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [pulsar] rdhabalia commented on issue #5513: Publish rate limit on broker to avoid OOM
Date Mon, 25 Nov 2019 22:39:51 GMT
rdhabalia commented on issue #5513: Publish rate limit on broker to avoid OOM
URL: https://github.com/apache/pulsar/issues/5513#issuecomment-558372993
 
 
   >> @merlimat 
   There's already a limit per-connection of max outstanding entries between broker and bookies.
Do you have a precise scenario for this to happen?
   
   >> @sijie 
   create a topic with 20 partitions. with parallelism == 20. The producer setting in the
pulsar sink is the default producer setting.
   
   
   I gave a try with similar setup and  somehow I couldn't reproduce it. and reason broker
doesn't go OOM because I think what @merlimat mentioned. Broker restricts max pending publish
request per connection.
   https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java#L130
   
   also, introducing counter across all topics for throttling can cause bottleneck while publishing
for all topics so, this feature might not be recommended for most of the users.
   
   so, I would recommend to depend on maxPendingRequestPerConnection rather adding more complexity
and if that's not working then it's worth to investigate why it disabling channel still cause
OOM. I have also created #5742 which can allow users to configure max-pending requests per
connection if needed.
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message