pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [pulsar] jiangmincong opened a new issue #8077: ERROR org.apache.bookkeeper.replication.ReplicationWorker - UnavailableException while replicating fragments
Date Thu, 17 Sep 2020 05:04:28 GMT

jiangmincong opened a new issue #8077:
URL: https://github.com/apache/pulsar/issues/8077


   **Describe the bug**
   A clear and concise description of what the bug is.
   21:28:02.513 [BookieJournal-3181] INFO  org.apache.bookkeeper.bookie.JournalChannel - Opening
journal /journal/bk-journal/current/17260dc8a5e.txn
   21:28:35.954 [GarbageCollectorThread-35-1] INFO  org.apache.bookkeeper.bookie.EntryLogCompactor
- Removing entry log 6058 after compaction
   21:28:36.128 [GarbageCollectorThread-35-1] INFO  org.apache.bookkeeper.bookie.GarbageCollectorThread
- Removing entry log metadata for 6058
   21:28:36.128 [GarbageCollectorThread-35-1] INFO  org.apache.bookkeeper.bookie.GarbageCollectorThread
- Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [0, 7, 2,
1, 1, 0, 0, 2, 2, 327]
   21:29:20.531 [main-SendThread(jxnc-bd-olmc-009:2181)] INFO  org.apache.zookeeper.ClientCnxn
- Unable to read additional data from server sessionid 0x902a3f7df0500d2, likely server has
closed socket, closing socket connection and attempting reconnect
   21:29:20.636 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase
- ZooKeeper client is disconnected from zookeeper now, but it is OK unless we received EXPIRED
event.
   21:29:21.547 [main-SendThread(jxnc-bd-olmc-006:2181)] INFO  org.apache.zookeeper.ClientCnxn
- Opening socket connection to server jxnc-bd-olmc-006/10.242.63.176:2181. Will not attempt
to authenticate using SASL (unknown error)
   21:29:21.548 [main-SendThread(jxnc-bd-olmc-006:2181)] INFO  org.apache.zookeeper.ClientCnxn
- Socket connection established, initiating session, client: /10.242.63.177:36774, server:
jxnc-bd-olmc-006/10.242.63.176:2181
   21:29:21.608 [main-SendThread(jxnc-bd-olmc-006:2181)] WARN  org.apache.zookeeper.ClientCnxn
- Unable to reconnect to ZooKeeper service, session 0x902a3f7df0500d2 has expired
   21:29:21.608 [main-EventThread] ERROR org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase
- ZooKeeper client connection to the ZooKeeper server has expired!
   21:29:21.608 [main-SendThread(jxnc-bd-olmc-006:2181)] INFO  org.apache.zookeeper.ClientCnxn
- Unable to reconnect to ZooKeeper service, session 0x902a3f7df0500d2 has expired, closing
socket connection
   21:29:21.608 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperClient -
ZooKeeper session 902a3f7df0500d2 is expired from jxnc-bd-olmc-006:2181,jxnc-bd-olmc-007:2181,jxnc-bd-olmc-008:2181,jxnc-bd-olmc-009:2181,jxnc-bd-olmc-010:2181.
   21:29:21.615 [main-EventThread] ERROR org.apache.bookkeeper.replication.AutoRecoveryMain
- Client connection to the Metadata server has expired, so shutting down AutoRecoveryMain!
   21:29:21.615 [main-EventThread] INFO  org.apache.bookkeeper.replication.AutoRecoveryMain
- Shutting down auto recovery: 3
   21:29:21.615 [main-EventThread] INFO  org.apache.bookkeeper.replication.AutoRecoveryMain
- Shutting down AutoRecovery
   21:29:21.669 [ZKC-connect-executor-0] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperClient
- Reconnecting zookeeper jxnc-bd-olmc-006:2181,jxnc-bd-olmc-007:2181,jxnc-bd-olmc-008:2181,jxnc-bd-olmc-009:2181,jxnc-bd-olmc-010:2181.
   21:29:21.683 [ZKC-connect-executor-0] INFO  org.apache.zookeeper.ZooKeeper - Initiating
client connection, connectString=jxnc-bd-olmc-006:2181,jxnc-bd-olmc-007:2181,jxnc-bd-olmc-008:2181,jxnc-bd-olmc-009:2181,jxnc-bd-olmc-010:2181
sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@420745d7
   21:29:21.683 [main-EventThread] INFO  org.apache.bookkeeper.replication.ReplicationWorker
- Shutting down replication worker
   21:29:21.683 [main-EventThread] INFO  org.apache.bookkeeper.replication.ReplicationWorker
- Shutting down ReplicationWorker
   21:29:21.683 [AuditorElector-10.242.63.177:3181] INFO  org.apache.bookkeeper.replication.AuditorElector
- Shutting down AuditorElector
   21:29:21.714 [ZKC-connect-executor-0] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer
value is 10485760 Bytes
   21:29:21.714 [ZKC-connect-executor-0] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout
value is 0. feature enabled=
   21:29:21.717 [ZKC-connect-executor-0-SendThread(jxnc-bd-olmc-007:2181)] INFO  org.apache.zookeeper.ClientCnxn
- Opening socket connection to server jxnc-bd-olmc-007/10.242.63.177:2181. Will not attempt
to authenticate using SASL (unknown error)
   21:29:21.717 [ZKC-connect-executor-0-SendThread(jxnc-bd-olmc-007:2181)] INFO  org.apache.zookeeper.ClientCnxn
- Socket connection established, initiating session, client: /10.242.63.177:33116, server:
jxnc-bd-olmc-007/10.242.63.177:2181
   21:29:21.718 [ZKC-connect-executor-0-SendThread(jxnc-bd-olmc-007:2181)] INFO  org.apache.zookeeper.ClientCnxn
- Session establishment complete on server jxnc-bd-olmc-007/10.242.63.177:2181, sessionid
= 0x702a3f516f40636, negotiated timeout = 30000
   21:29:21.718 [ZKC-connect-executor-0-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase
- ZooKeeper client is connected now.
   21:29:21.718 [ZKC-connect-executor-0] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperClient
- ZooKeeper session 702a3f516f40636 is created to jxnc-bd-olmc-006:2181,jxnc-bd-olmc-007:2181,jxnc-bd-olmc-008:2181,jxnc-bd-olmc-009:2181,jxnc-bd-olmc-010:2181.
   21:29:21.717 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker
- UnavailableException while replicating fragments
   org.apache.bookkeeper.replication.ReplicationException$UnavailableException: Interrupted
while contacting zookeeper
   	at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.isLedgerReplicationEnabled(ZkLedgerUnderreplicationManager.java:732)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.waitIfLedgerReplicationDisabled(ZkLedgerUnderreplicationManager.java:619)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicate(ZkLedgerUnderreplicationManager.java:600)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:272)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:238)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
   	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
   Caused by: java.lang.InterruptedException
   	at java.lang.Object.wait(Native Method) ~[?:1.8.0_192]
   	at java.lang.Object.wait(Object.java:502) ~[?:1.8.0_192]
   	at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) ~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   	at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) ~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:2016) ~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2301(ZooKeeperClient.java:70)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$13.call(ZooKeeperClient.java:830) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$13.call(ZooKeeperClient.java:824) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient.exists(ZooKeeperClient.java:824) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:2049) ~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2401(ZooKeeperClient.java:70)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$14.call(ZooKeeperClient.java:851) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$14.call(ZooKeeperClient.java:845) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient.exists(ZooKeeperClient.java:845) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.isLedgerReplicationEnabled(ZkLedgerUnderreplicationManager.java:723)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	... 6 more
   21:29:21.745 [ReplicationWorker] INFO  org.apache.bookkeeper.replication.ReplicationWorker
- ReplicationWorker exited loop!
   21:29:21.756 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.179:3181
   21:29:21.777 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.199:3181
   21:29:21.779 [bookkeeper-io-50-52] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0xcf0cda8c, L:/10.242.63.177:52874 ! R:/10.242.63.179:3181]
   21:29:21.779 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.177:3181
   21:29:21.779 [bookkeeper-io-50-50] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x4d645ee5, L:/10.242.63.177:53768 ! R:/10.242.63.199:3181]
   21:29:21.781 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.178:3181
   21:29:21.781 [bookkeeper-io-50-45] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0xbcd90045, L:/10.242.63.177:51154 ! R:/10.242.63.177:3181]
   21:29:21.783 [bookie-io-1-33] INFO  org.apache.bookkeeper.proto.BookieRequestHandler -
Channels disconnected: [id: 0x1a719422, L:/10.242.63.177:3181 ! R:/10.242.63.177:51154]
   21:29:21.783 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.175:3181
   21:29:21.783 [bookkeeper-io-50-44] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x79a921b7, L:/10.242.63.177:55774 ! R:/10.242.63.178:3181]
   21:29:21.785 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.197:3181
   21:29:21.785 [bookkeeper-io-50-76] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0xedd7ec8a, L:/10.242.63.177:50894 ! R:/10.242.63.175:3181]
   21:29:21.786 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.198:3181
   21:29:21.786 [bookkeeper-io-50-40] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x9eef5b31, L:/10.242.63.177:44234 ! R:/10.242.63.197:3181]
   21:29:21.788 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.176:3181
   21:29:21.789 [bookkeeper-io-50-46] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x8b324bea, L:/10.242.63.177:36426 ! R:jxnc-bd-olmc-028/10.242.63.198:3181]
   21:29:21.789 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.195:3181
   21:29:21.789 [bookkeeper-io-50-60] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x6505713a, L:/10.242.63.177:54268 ! R:/10.242.63.176:3181]
   21:29:21.791 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.173:3181
   21:29:21.791 [bookkeeper-io-50-37] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x259ea28e, L:/10.242.63.177:50694 ! R:/10.242.63.195:3181]
   21:29:21.793 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.196:3181
   21:29:21.793 [bookkeeper-io-50-83] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x564ddd2e, L:/10.242.63.177:38982 ! R:jxnc-bd-olmc-003/10.242.63.173:3181]
   21:29:21.794 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.174:3181
   21:29:21.794 [bookkeeper-io-50-39] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x2967b15b, L:/10.242.63.177:46508 ! R:/10.242.63.196:3181]
   21:29:21.796 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.171:3181
   21:29:21.796 [bookkeeper-io-50-58] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x0255fa8b, L:/10.242.63.177:37096 ! R:/10.242.63.174:3181]
   21:29:21.797 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.172:3181
   21:29:21.797 [bookkeeper-io-50-59] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0xb0f71766, L:/10.242.63.177:56610 ! R:/10.242.63.171:3181]
   21:29:21.799 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.194:3181
   21:29:21.800 [bookkeeper-io-50-43] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x744b9bc0, L:/10.242.63.177:34736 ! R:/10.242.63.172:3181]
   21:29:21.800 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.191:3181
   21:29:21.800 [bookkeeper-io-50-57] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0xdf0ae60a, L:/10.242.63.177:57296 ! R:/10.242.63.194:3181]
   21:29:21.802 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.192:3181
   21:29:21.802 [bookkeeper-io-50-54] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x2d8f2f79, L:/10.242.63.177:33988 ! R:/10.242.63.191:3181]
   21:29:21.804 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.190:3181
   21:29:21.807 [bookkeeper-io-50-41] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x3d27521e, L:/10.242.63.177:42342 ! R:/10.242.63.192:3181]
   21:29:21.807 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.188:3181
   21:29:21.807 [bookkeeper-io-50-55] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0xacc73cb1, L:/10.242.63.177:35546 ! R:/10.242.63.190:3181]
   21:29:21.809 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.200:3181
   21:29:21.809 [bookkeeper-io-50-56] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x3ab7591d, L:/10.242.63.177:55812 ! R:/10.242.63.188:3181]
   21:29:21.810 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.189:3181
   21:29:21.810 [bookkeeper-io-50-47] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x6e9f7b8d, L:/10.242.63.177:52520 ! R:/10.242.63.200:3181]
   21:29:21.812 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.186:3181
   21:29:21.812 [bookkeeper-io-50-48] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x70bf160e, L:/10.242.63.177:52438 ! R:/10.242.63.189:3181]
   21:29:21.813 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.187:3181
   21:29:21.813 [bookkeeper-io-50-35] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x778a8f55, L:/10.242.63.177:40600 ! R:jxnc-bd-olmc-016/10.242.63.186:3181]
   21:29:21.813 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.184:3181
   21:29:21.813 [bookkeeper-io-50-36] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x6271e956, L:/10.242.63.177:59924 ! R:jxnc-bd-olmc-017/10.242.63.187:3181]
   21:29:21.815 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.185:3181
   21:29:21.815 [bookkeeper-io-50-49] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0xd38880f1, L:/10.242.63.177:37420 ! R:/10.242.63.184:3181]
   21:29:21.817 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.182:3181
   21:29:21.817 [bookkeeper-io-50-51] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x59002b72, L:/10.242.63.177:46156 ! R:jxnc-bd-olmc-015/10.242.63.185:3181]
   21:29:21.818 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.183:3181
   21:29:21.818 [bookkeeper-io-50-61] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x066deb9c, L:/10.242.63.177:53474 ! R:/10.242.63.182:3181]
   21:29:21.819 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.180:3181
   21:29:21.819 [bookkeeper-io-50-53] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0xed6e3cc6, L:/10.242.63.177:39112 ! R:/10.242.63.183:3181]
   21:29:21.821 [main-EventThread] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for 10.242.63.181:3181
   21:29:21.821 [bookkeeper-io-50-38] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0x102b4a79, L:/10.242.63.177:60566 ! R:jxnc-bd-olmc-010/10.242.63.180:3181]
   21:29:21.821 [bookkeeper-io-50-42] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient
- Disconnected from bookie channel [id: 0xbb2b5056, L:/10.242.63.177:46174 ! R:/10.242.63.181:3181]
   21:29:21.887 [AuditorElector-10.242.63.177:3181] ERROR org.apache.bookkeeper.replication.AuditorElector
- Exception while deleting myVote:/ledgers/underreplication/auditorelection/V_0000000169
   org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session
expired for /ledgers/underreplication/auditorelection/V_0000000169
   	at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) ~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1793) ~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2101(ZooKeeperClient.java:70)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$11.call(ZooKeeperClient.java:770) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$11.call(ZooKeeperClient.java:764) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient.delete(ZooKeeperClient.java:764) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.replication.AuditorElector$2.run(AuditorElector.java:262) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192]
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
   	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
   21:29:21.950 [main-EventThread] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x702a3f516f40636
closed
   21:29:21.950 [main-EventThread] INFO  org.apache.bookkeeper.meta.AbstractZkLedgerManager
- ZooKeeper client expired on ledger manager.
   21:29:21.950 [ZKC-connect-executor-0-EventThread] INFO  org.apache.zookeeper.ClientCnxn
- EventThread shut down for session: 0x702a3f516f40636
   21:29:21.952 [main-EventThread] WARN  org.apache.bookkeeper.discover.ZKRegistrationClient
- Failed to schedule watch bookies task
   java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@522f30a
rejected from java.util.concurrent.ScheduledThreadPoolExecutor@6ac3341d[Terminated, pool size
= 0, active threads = 0, queued tasks = 0, completed tasks = 2749179]
   	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
~[?:1.8.0_192]
   	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_192]
   	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
~[?:1.8.0_192]
   	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
~[?:1.8.0_192]
   	at com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator.schedule(MoreExecutors.java:555)
~[com.google.guava-guava-25.1-jre.jar:?]
   	at org.apache.bookkeeper.common.util.BoundedScheduledExecutorService.schedule(BoundedScheduledExecutorService.java:63)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.common.util.OrderedScheduler$OrderedSchedulerDecoratedThread.schedule(OrderedScheduler.java:317)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.common.util.OrderedScheduler.schedule(OrderedScheduler.java:273)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.discover.ZKRegistrationClient$WatchTask.scheduleWatchTask(ZKRegistrationClient.java:107)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.discover.ZKRegistrationClient$WatchTask.process(ZKRegistrationClient.java:149)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) [org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   21:29:21.953 [main-EventThread] WARN  org.apache.bookkeeper.discover.ZKRegistrationClient
- Failed to schedule watch bookies task
   java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@31652151
rejected from java.util.concurrent.ScheduledThreadPoolExecutor@6ac3341d[Terminated, pool size
= 0, active threads = 0, queued tasks = 0, completed tasks = 2749179]
   	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
~[?:1.8.0_192]
   	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_192]
   	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
~[?:1.8.0_192]
   	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
~[?:1.8.0_192]
   	at com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator.schedule(MoreExecutors.java:555)
~[com.google.guava-guava-25.1-jre.jar:?]
   	at org.apache.bookkeeper.common.util.BoundedScheduledExecutorService.schedule(BoundedScheduledExecutorService.java:63)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.common.util.OrderedScheduler$OrderedSchedulerDecoratedThread.schedule(OrderedScheduler.java:317)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.common.util.OrderedScheduler.schedule(OrderedScheduler.java:273)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.discover.ZKRegistrationClient$WatchTask.scheduleWatchTask(ZKRegistrationClient.java:107)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.discover.ZKRegistrationClient$WatchTask.process(ZKRegistrationClient.java:149)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) [org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   21:29:21.953 [main-EventThread] ERROR org.apache.bookkeeper.replication.AuditorElector
- Lost ZK connection, shutting down
   21:29:21.953 [main-EventThread] ERROR org.apache.zookeeper.ClientCnxn - Error while calling
watcher 
   java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6856522d
rejected from java.util.concurrent.ThreadPoolExecutor@7afe568e[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 2]
   	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
~[?:1.8.0_192]
   	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_192]
   	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_192]
   	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
~[?:1.8.0_192]
   	at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
~[?:1.8.0_192]
   	at org.apache.bookkeeper.replication.AuditorElector.submitShutdownTask(AuditorElector.java:254)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.replication.AuditorElector.access$100(AuditorElector.java:80)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.bookkeeper.replication.AuditorElector$ElectionWatcher.process(AuditorElector.java:238)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) [org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
   21:29:21.953 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut
down for session: 0x902a3f7df0500d2
   21:29:22.383 [AutoRecoveryDeathWatcher-3181] INFO  org.apache.bookkeeper.replication.AutoRecoveryMain
- AutoRecoveryDeathWatcher noticed the AutoRecovery is not running any more,exiting the watch
loop!
   21:29:22.383 [AutoRecoveryDeathWatcher-3181] ERROR org.apache.bookkeeper.common.component.ComponentStarter
- Triggered exceptionHandler of Component: bookie-server because of Exception in Thread: Thread[AutoRecoveryDeathWatcher-3181,5,main]
   java.lang.RuntimeException: AutoRecovery is not running any more
   	at org.apache.bookkeeper.replication.AutoRecoveryMain$AutoRecoveryDeathWatcher.run(AutoRecoveryMain.java:237)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
   21:29:22.385 [component-shutdown-thread] INFO  org.apache.bookkeeper.common.component.ComponentStarter
- Closing component bookie-server in shutdown hook.
   21:29:22.387 [component-shutdown-thread] INFO  org.apache.bookkeeper.replication.AutoRecoveryMain
- Shutting down auto recovery: 0
   21:29:22.387 [component-shutdown-thread] INFO  org.apache.bookkeeper.proto.BookieServer
- Shutting down BookieServer
   21:29:22.387 [component-shutdown-thread] INFO  org.apache.bookkeeper.proto.BookieNettyServer
- Shutting down BookieNettyServer
   **To Reproduce**
   Steps to reproduce the behavior:
   1. Go to '...'
   2. Click on '....'
   3. Scroll down to '....'
   4. See error
   
   **Expected behavior**
   A clear and concise description of what you expected to happen.
   
   **Screenshots**
   If applicable, add screenshots to help explain your problem.
   
   **Desktop (please complete the following information):**
    - OS: [e.g. iOS]
   
   **Additional context**
   Add any other context about the problem here.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message