pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] bardock commented on issue #2431: Unable to consume messages, need to re-subscribe client
Date Wed, 06 Feb 2019 16:14:12 GMT
bardock commented on issue #2431: Unable to consume messages, need to re-subscribe client
URL: https://github.com/apache/pulsar/issues/2431#issuecomment-461082870
 
 
   Hello @merlimat @sijie,
   
   Yesterday, we faced this problem again.
   Here are the stats: 
   [topic-stats.txt](https://github.com/apache/pulsar/files/2837146/topic-stats.txt)
   [topic-stats-internal.txt](https://github.com/apache/pulsar/files/2837147/topic-stats-internal.txt)
   
   Brokers were logging this:
   
   ```
   2019-02-05 17:25:38,735 - [level:WARN] [class:ManagedCursorImpl$11] [line:958] [thread:BookKeeperClientWorker-20-1]
- [fury/global/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news/persistent/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news-partition-3][f6f49168781f402c99ddfa871bc0e90c-fury-news-sender-ios-meli.notifications-providers]
Error while replaying entries
   ```
   
   ```
   2019-02-05 17:25:38,735 - [level:ERROR] [class:PersistentDispatcherMultipleConsumers] [line:399]
[thread:BookKeeperClientWorker-20-1] - [persistent://fury/global/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news-partition-3
/ f6f49168781f402c99ddfa871bc0e90c-fury-news-sender-ios-meli.notifications-providers] Error
reading entries at 1666190:133506 : org.apache.bookkeeper.client.BKException$BKTooManyRequestsException,
Read Type Replay - Retrying to read in 15.0 seconds
   ```
   
   ```
   2019-02-05 17:25:40,713 - [level:ERROR] [class:PersistentDispatcherMultipleConsumers] [line:399]
[thread:BookKeeperClientWorker-20-1] - [persistent://fury/global/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news-partition-3
/ f6f49168781f402c99ddfa871bc0e90c-fury-news-sender-ios-meli.notifications-providers] Error
reading entries at 1666190:133506 : org.apache.bookkeeper.client.BKException$BKTimeoutException,
Read Type Replay - Retrying to read in 29.185 seconds
   ```
   
   A bookie was also with higher CPU than the rest (~60%). We restarted some bookies and every
broker and the problem persisted but some minutes later the problem was solved.
   
   ![image](https://user-images.githubusercontent.com/1980715/52353590-a2748f00-2a0d-11e9-9648-d17cb50710cb.png)
   As you can see in the above image, at around 14:00 started the high cpu in that bookie,
we restarted it at around 14:20 and a few minutes later the cpu was high again but this time
the bookie has few or none read/write activity.
   
   We believe we hit this bug: https://github.com/apache/bookkeeper/pull/1620
   So we are planning to update to the last stable version (4.7.3).
   
   As you can see in internal stats, subscription `f6f49168781f402c99ddfa871bc0e90c-fury-news-sender-ios-meli.notifications-providers`
had some weird values:
   ```
         "messagesConsumedCounter" : -116637,
         "cursorLedger" : -1,
         "cursorLedgerLastEntry" : -1,
         "state" : "NoLedger"
   ```
   
   Is that cursor's state expected?
   
   Thanks!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message