pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] rdhabalia opened a new pull request #3347: Support Add-entry timeout at broker to avoid stuck topics
Date Thu, 10 Jan 2019 01:01:52 GMT
rdhabalia opened a new pull request #3347: Support Add-entry timeout at broker to avoid stuck
topics
URL: https://github.com/apache/pulsar/pull/3347
 
 
   ### Motivation
   
   Recently and in past, we have seen few instances where bookie crashes when it goes out
of memory and kernel panics and broker doesn't complete add-entry callback because of that
topics get stuck on pending-writes.
   
   ```
   2010-01-01T00:18:19+0000 2010-01-01 00:18:19,594 21907 report_stuck:556 WARNING persistent://prop1/global/prod-rt-mxt/extractions-partition-0
is stuck (broker1.us-west1.com): waiting for write
   2010-01-01T00:18:22+0000 2010-01-01 00:18:22,654 21907 report_stuck:556 WARNING persistent://prop1/global/mbr-events/27_bf1
is stuck (broker1.us-west1.com): waiting for write
   2010-01-01T00:19:10+0000 2010-01-01 00:19:10,790 21907 report_stuck:556 WARNING persistent://prop1/global/mbr-events/30_bf1
is stuck (broker1.us-west1.com): waiting for write
   2010-01-01T00:19:14+0000 2010-01-01 00:19:14,692 21907 report_stuck:556 WARNING persistent://prop1/global/mbr-events/35_gq2
is stuck (broker1.us-west1.com): waiting for write
   2010-01-01T00:19:15+0000 2010-01-01 00:19:15,809 21907 report_stuck:556 WARNING persistent://prop1/global/jedi-events/batchevents-partition-35
is stuck (broker1.us-west1.com): waiting for write
   2010-01-01T00:19:25+0000 2010-01-01 00:19:25,127 21907 report_stuck:556 WARNING persistent://prop1/global/jedi-events/mailevents-partition-24
is stuck (broker1.us-west1.com): waiting for write
   ```
   
   internal-stats
   ```
   {
     "entriesAddedCounter" : 66066317,
     "numberOfEntries" : 14129359,
     "totalSize" : 145505085493,
     "currentLedgerEntries" : 238938,
     "currentLedgerSize" : 3715327400,
     "lastLedgerCreatedTimestamp" : "2018-11-27 00:03:03.125+0000",
     "waitingCursorsCount" : 0,
     "pendingAddEntriesCount" : 252514,
     "lastConfirmedEntry" : "489781963:26121",
     "state" : "ClosingLedger",
     "ledgers" : [ {
       "ledgerId" : 489765238,
       "entries" : 559855,
       "size" : 6928710817
     }, {
   ...
   ```
   
   ### Modifications
   
   Add support of write timeout which can be disable by configuring timeout=0.
   
   ### Result
   
   topic will not stuck in pending write in case add-entry callback doesn't completes.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message