pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] nkurihar commented on issue #3535: Broker stopped with a lot of LedgerFencedException
Date Thu, 07 Feb 2019 04:22:40 GMT
nkurihar commented on issue #3535: Broker stopped with a lot of LedgerFencedException
URL: https://github.com/apache/pulsar/issues/3535#issuecomment-461283589
 
 
   @sijie Thanks for your help!
   
   > I think the exception should be self-recovered. did it self-recover?
   
   No.
   After ZK session expired, Broker becoame down.
   ```
   01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-SendThread(zk101.pulsar.xxx.xx:2181)]
WARN  org.apache.zookeeper.ClientCnxn      - Unable to reconnect to ZooKeeper service, session
0x3048c994034e22a has expired
   01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-SendThread(zk101.pulsar.xxx.xx:2181)]
INFO  org.apache.zookeeper.ClientCnxn      - Unable to reconnect to ZooKeeper service, session
0x3048c994034e22a has expired, closing socket connection
   01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.zookeeper.ZooKeeperDataCache
  - [State:CLOSED sessionid:0x3048c994034e22a local:/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181
lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1]
Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
   01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN  o.a.p.zookeeper.LocalZooKeeperCache
 - Lost connection from local ZK. Invalidating the whole cache.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.z.ZooKeeperChildrenCache
      - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181
lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1]
Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN  o.a.p.zookeeper.LocalZooKeeperCache
 - Lost connection from local ZK. Invalidating the whole cache.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.z.ZooKeeperChildrenCache
      - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181
lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1]
Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN  o.a.p.zookeeper.LocalZooKeeperCache
 - Lost connection from local ZK. Invalidating the whole cache.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.zookeeper.ZooKeeperDataCache
  - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181
lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1]
Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN  o.a.p.zookeeper.LocalZooKeeperCache
 - Lost connection from local ZK. Invalidating the whole cache.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.zookeeper.ZooKeeperDataCache
  - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181
lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1]
Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN  o.a.p.zookeeper.LocalZooKeeperCache
 - Lost connection from local ZK. Invalidating the whole cache.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] ERROR o.a.b.zookeeper.ZooKeeperWatcherBase
- ZooKeeper client connection to the ZooKeeper server has expired!
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.b.zookeeper.ZooKeeperClient
     - ZooKeeper session 3048c994034e22a is expired from zk100.pulsar.xxx.xx,zk101.pulsar.xxx.xx,zk102.pulsar.xxx.xx,zk103.pulsar.xxx.xx,zk104.pulsar.xxx.xx.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.z.ZooKeeperSessionWatcher
     - Received zookeeper notification, eventType=None, eventState=Expired
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] ERROR o.a.p.z.ZooKeeperSessionWatcher
     - ZooKeeper session already expired, invoking shutdown
   01:47:09.963 [pulsar-zk-session-watcher-5-1] INFO  o.a.p.z.ZooKeeperSessionWatcher    
 - ZooKeeperSessionWatcher interrupted
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.b.MessagingServiceShutdownHook
- Invoking Runtime.halt(-1)
   01:47:09.964 [ZKC-connect-executor-0] INFO  o.a.b.zookeeper.ZooKeeperClient      - Reconnecting
zookeeper zk100.pulsar.xxx.xx,zk101.pulsar.xxx.xx,zk102.pulsar.xxx.xx,zk103.pulsar.xxx.xx,zk104.pulsar.xxx.xx.
   01:47:09.966 [ZKC-connect-executor-0] INFO  org.apache.zookeeper.ZooKeeper       - Initiating
client connection, connectString=zk100.pulsar.xxx.xx,zk101.pulsar.xxx.xx,zk102.pulsar.xxx.xx,zk103.pulsar.xxx.xx,zk104.pulsar.xxx.xx
sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@1c7f3bee
   
   (Broker became down and not self-recovered)
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message