rdhabalia commented on issue #5513: Publish rate limit on broker to avoid OOM
URL: https://github.com/apache/pulsar/issues/5513#issuecomment-558372993
>> @merlimat
There's already a limit per-connection of max outstanding entries between broker and bookies.
Do you have a precise scenario for this to happen?
>> @sijie
create a topic with 20 partitions. with parallelism == 20. The producer setting in the
pulsar sink is the default producer setting.
I gave a try with similar setup and somehow I couldn't reproduce it. and reason broker
doesn't go OOM because I think what @merlimat mentioned. Broker restricts max pending publish
request per connection.
https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java#L130
also, introducing counter across all topics for throttling can cause bottleneck while publishing
for all topics so, this feature might not be recommended for most of the users.
so, I would recommend to depend on maxPendingRequestPerConnection rather adding more complexity
and if that's not working then it's worth to investigate why it disabling channel still cause
OOM. I have also created #5742 which can allow users to configure max-pending requests per
connection if needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
|