bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Kebinger <jkebin...@hubspot.com>
Subject Re: Synchronous send blocked forever by changing ensemble.
Date Wed, 02 Oct 2013 12:25:34 GMT
Thanks Sijie, we'll look forward to installing 4.2.2 . In the meantime
we're replacing our synchronous call with an asynchronous call combined
with a countdown latch and timeout so we can throw away the client instance
if this happens again.


On Tue, Oct 1, 2013 at 7:24 PM, Sijie Guo <guosijie@gmail.com> wrote:

>
>
>
> On Tue, Oct 1, 2013 at 1:51 PM, James Kebinger <jkebinger@hubspot.com>wrote:
>
>> We're doing synchronous writes to BK using the addEntry method on
>> LedgerHandle. There's no way to set a timeout on this call, so we're having
>> threads blocked "forever" (we have to bounce API nodes to clear them) while
>> the bookkeeper client seems to spin its wheels connecting to various bk
>> servers and changing the ensemble. I'll paste some logs below. More
>> available upon request :)
>>
>> Two questions:
>> 1) Should the synchronous method have an optional timeout so we can take
>> measures to instantiate a new bookkeeper client?
>>
>> 2) Given the logs below, can anyone figure out why the reconnects didn't
>> eventually find a live BK? During this time other services were writing
>> successfully to the same book keeper cluster.
>>
>>
> Hmm. It looks like that you are hitting the race condition :
> BOOKKEEPER-620 <https://issues.apache.org/jira/browse/BOOKKEEPER-620> .
> The newly connected connection is disconnected by previous disconnect
> event, which generate disconnect event and disconnect following new
> connections. So the ledger ends up changing bookies.
>
> We are going to release 4.2.2 including this fix recently (properly this
> week). If it is urgent, you could properly apply the patch in
> BOOKKKEEPER-620 <https://issues.apache.org/jira/browse/BOOKKEEPER-620>first.
>
> For timeout, we could consider adding timeout for synchronous methods. If
> you'd like, you could make a change for it. Or, I could create a jira for
> this work.
>
> -Sijie
>
>
>
>
>>
>> We're on bookkeeper 4.2.1
>>
>> INFO  [2013-09-30 15:47:55,172] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,172] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,172] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,172] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 2
>> ERROR [2013-09-30 15:47:55,172] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,175] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,177] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,181] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
>> ERROR [2013-09-30 15:47:55,181] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
>> WARN  [2013-09-30 15:47:55,181] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,181] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
>> ERROR [2013-09-30 15:47:55,181] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,181] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.145.202.79:3181, /10.72.43.204:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   ensembleMember: "10.72.43.204:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: ""
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.145.202.79:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: ""
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,187] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,188] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,190] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,191] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,191] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,191] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,191] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
>> ERROR [2013-09-30 15:47:55,191] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,191] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,194] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,194] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
>> WARN  [2013-09-30 15:47:55,194] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,194] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 2
>> ERROR [2013-09-30 15:47:55,194] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,200] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,202] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,205] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
>> ERROR [2013-09-30 15:47:55,205] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
>> WARN  [2013-09-30 15:47:55,205] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,205] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
>> ERROR [2013-09-30 15:47:55,205] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,206] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.145.202.79:3181, /10.2.9.59:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.145.202.79:3181"
>>   ensembleMember: "10.72.43.204:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,211] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,212] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,215] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
>> ERROR [2013-09-30 15:47:55,216] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
>> WARN  [2013-09-30 15:47:55,216] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,216] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 1
>> ERROR [2013-09-30 15:47:55,216] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,216] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,217] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,219] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,219] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,219] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,219] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 2
>> ERROR [2013-09-30 15:47:55,219] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,222] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,223] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,227] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
>> ERROR [2013-09-30 15:47:55,227] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
>> WARN  [2013-09-30 15:47:55,227] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,227] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
>> ERROR [2013-09-30 15:47:55,227] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,231] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.145.202.79:3181, /10.232.13.238:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.72.43.204:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.145.202.79:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,233] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,234] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,237] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,237] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
>> WARN  [2013-09-30 15:47:55,237] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,237] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
>> ERROR [2013-09-30 15:47:55,237] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,240] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.232.13.238:3181
>> INFO  [2013-09-30 15:47:55,241] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.232.13.238:3181
>> INFO  [2013-09-30 15:47:55,243] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,243] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.232.13.238:3181
>> ERROR [2013-09-30 15:47:55,243] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
>> WARN  [2013-09-30 15:47:55,243] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,243] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.232.13.238:3181 index: 2
>> ERROR [2013-09-30 15:47:55,243] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,244] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,246] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,246] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,246] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
>> ERROR [2013-09-30 15:47:55,247] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,247] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,250] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,251] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
>> ERROR [2013-09-30 15:47:55,254] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
>> WARN  [2013-09-30 15:47:55,254] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,254] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 2
>> ERROR [2013-09-30 15:47:55,254] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,255] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,260] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.232.13.238:3181, /10.169.0.213:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   ensembleMember: "10.144.81.168:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,261] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,263] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,266] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
>> ERROR [2013-09-30 15:47:55,267] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
>> WARN  [2013-09-30 15:47:55,267] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,267] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 2
>> ERROR [2013-09-30 15:47:55,267] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,271] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.232.13.238:3181
>> INFO  [2013-09-30 15:47:55,272] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.232.13.238:3181
>> INFO  [2013-09-30 15:47:55,272] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,274] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,276] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.232.13.238:3181
>> ERROR [2013-09-30 15:47:55,276] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
>> WARN  [2013-09-30 15:47:55,276] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,276] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.232.13.238:3181 index: 1
>> ERROR [2013-09-30 15:47:55,276] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> ERROR [2013-09-30 15:47:55,278] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
>> WARN  [2013-09-30 15:47:55,278] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,278] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 2
>> INFO  [2013-09-30 15:47:55,278] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
>> ERROR [2013-09-30 15:47:55,278] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,281] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,282] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,286] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,286] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,286] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
>> INFO  [2013-09-30 15:47:55,286] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,286] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,289] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.2.9.59:3181, /10.72.43.204:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.144.81.168:3181"
>>   ensembleMember: "10.72.43.204:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   ensembleMember: "10.145.202.79:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,318] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,320] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
>> ERROR [2013-09-30 15:47:55,324] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
>> WARN  [2013-09-30 15:47:55,324] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,324] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 1
>> INFO  [2013-09-30 15:47:55,324] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
>> ERROR [2013-09-30 15:47:55,324] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,326] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,326] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,329] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,329] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
>> WARN  [2013-09-30 15:47:55,329] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,329] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 2
>> ERROR [2013-09-30 15:47:55,329] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,330] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,332] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
>> ERROR [2013-09-30 15:47:55,335] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
>> WARN  [2013-09-30 15:47:55,335] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,335] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
>> INFO  [2013-09-30 15:47:55,335] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
>> ERROR [2013-09-30 15:47:55,335] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,342] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,342] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,342] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,344] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,345] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,345] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
>> WARN  [2013-09-30 15:47:55,346] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,346] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
>> ERROR [2013-09-30 15:47:55,346] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> ERROR [2013-09-30 15:47:55,348] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
>> WARN  [2013-09-30 15:47:55,348] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,348] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 2
>> INFO  [2013-09-30 15:47:55,348] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
>> ERROR [2013-09-30 15:47:55,348] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,352] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,353] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,355] com.hubspot.connect.notify.SentryClient: Got response code from Sentry notify: 429 - UNKNOWN STATUS CODE
>> INFO  [2013-09-30 15:47:55,355] com.hubspot.connect.notify.SentryClient: Sentry responded to a 124722 character msg in 253 ms unsuccessfully
>> INFO  [2013-09-30 15:47:55,357] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,357] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,357] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,357] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
>> ERROR [2013-09-30 15:47:55,357] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,358] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.2.9.59:3181, /10.232.13.238:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.145.202.79:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   ensembleMember: "10.144.81.168:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,362] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,364] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,368] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
>> ERROR [2013-09-30 15:47:55,368] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
>> WARN  [2013-09-30 15:47:55,368] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,368] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
>> ERROR [2013-09-30 15:47:55,369] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,369] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.232.13.238:3181
>> INFO  [2013-09-30 15:47:55,370] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.232.13.238:3181
>> INFO  [2013-09-30 15:47:55,373] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.232.13.238:3181
>> ERROR [2013-09-30 15:47:55,374] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
>> WARN  [2013-09-30 15:47:55,374] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,374] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.232.13.238:3181 index: 2
>> ERROR [2013-09-30 15:47:55,374] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,375] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,375] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,378] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,378] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
>> WARN  [2013-09-30 15:47:55,378] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,378] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
>> ERROR [2013-09-30 15:47:55,378] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,384] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,385] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,385] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,387] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,388] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,388] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,388] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,388] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
>> ERROR [2013-09-30 15:47:55,389] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> ERROR [2013-09-30 15:47:55,390] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
>> WARN  [2013-09-30 15:47:55,390] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,390] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,390] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 2
>> ERROR [2013-09-30 15:47:55,390] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,394] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,395] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,397] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,397] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
>> WARN  [2013-09-30 15:47:55,397] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,398] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
>> ERROR [2013-09-30 15:47:55,398] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,399] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.72.43.204:3181, /10.144.81.168:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.144.81.168:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.72.43.204:3181"
>>   ensembleMember: "10.145.202.79:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,404] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,405] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
>> ERROR [2013-09-30 15:47:55,408] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
>> WARN  [2013-09-30 15:47:55,408] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,408] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 1
>> INFO  [2013-09-30 15:47:55,408] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
>> ERROR [2013-09-30 15:47:55,408] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,411] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,413] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,413] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,414] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,417] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,417] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
>> WARN  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 2
>> INFO  [2013-09-30 15:47:55,417] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,417] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> ERROR [2013-09-30 15:47:55,417] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
>> WARN  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
>> ERROR [2013-09-30 15:47:55,418] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,423] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,424] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,427] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.232.13.238:3181, /10.2.9.59:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.72.43.204:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,427] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,427] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,427] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,427] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 2
>> ERROR [2013-09-30 15:47:55,428] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,434] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.232.13.238:3181
>> INFO  [2013-09-30 15:47:55,435] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.232.13.238:3181
>> ERROR [2013-09-30 15:47:55,438] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
>> WARN  [2013-09-30 15:47:55,438] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,438] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.232.13.238:3181 index: 1
>> INFO  [2013-09-30 15:47:55,438] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.232.13.238:3181
>> ERROR [2013-09-30 15:47:55,438] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,440] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.232.13.238:3181, /10.169.0.213:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.72.43.204:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,444] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,445] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,448] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,448] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
>> WARN  [2013-09-30 15:47:55,448] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,448] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
>> ERROR [2013-09-30 15:47:55,448] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,454] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,455] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,455] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,457] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,458] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,458] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,458] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,458] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
>> ERROR [2013-09-30 15:47:55,459] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,460] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
>> ERROR [2013-09-30 15:47:55,460] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
>> WARN  [2013-09-30 15:47:55,460] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,460] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 2
>> ERROR [2013-09-30 15:47:55,460] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,464] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,466] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,469] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.144.81.168:3181, /10.2.9.59:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.144.81.168:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.144.81.168:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> ERROR [2013-09-30 15:47:55,470] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
>> WARN  [2013-09-30 15:47:55,470] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,470] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 1
>> INFO  [2013-09-30 15:47:55,470] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
>> ERROR [2013-09-30 15:47:55,470] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,474] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,475] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,477] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,477] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,477] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,477] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 2
>> ERROR [2013-09-30 15:47:55,477] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,478] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.145.202.79:3181, /10.2.9.59:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.145.202.79:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.144.81.168:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,483] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,484] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,486] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,486] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
>> ERROR [2013-09-30 15:47:55,486] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
>> WARN  [2013-09-30 15:47:55,486] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,486] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 2
>> ERROR [2013-09-30 15:47:55,486] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,488] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,492] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
>> ERROR [2013-09-30 15:47:55,492] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
>> WARN  [2013-09-30 15:47:55,492] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,492] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
>> ERROR [2013-09-30 15:47:55,492] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,492] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.232.13.238:3181
>> INFO  [2013-09-30 15:47:55,493] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.232.13.238:3181
>> ERROR [2013-09-30 15:47:55,496] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
>> WARN  [2013-09-30 15:47:55,496] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,496] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.232.13.238:3181 index: 2
>> INFO  [2013-09-30 15:47:55,496] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.232.13.238:3181
>> ERROR [2013-09-30 15:47:55,496] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,502] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,503] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,503] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,505] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,507] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
>> ERROR [2013-09-30 15:47:55,507] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
>> WARN  [2013-09-30 15:47:55,508] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,508] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 2
>> ERROR [2013-09-30 15:47:55,508] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,508] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,510] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,510] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,510] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
>> ERROR [2013-09-30 15:47:55,511] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,513] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,514] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,519] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
>> ERROR [2013-09-30 15:47:55,519] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
>> WARN  [2013-09-30 15:47:55,519] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,519] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 2
>> ERROR [2013-09-30 15:47:55,519] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,524] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.145.202.79:3181, /10.169.0.213:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.145.202.79:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,527] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,527] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,530] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,530] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,530] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,530] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 2
>> ERROR [2013-09-30 15:47:55,530] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,536] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,536] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,537] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,538] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
>> INFO  [2013-09-30 15:47:55,539] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,539] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
>> WARN  [2013-09-30 15:47:55,539] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,539] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 2
>> ERROR [2013-09-30 15:47:55,539] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,542] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
>> ERROR [2013-09-30 15:47:55,542] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
>> WARN  [2013-09-30 15:47:55,542] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,543] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
>> ERROR [2013-09-30 15:47:55,543] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,546] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,548] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
>> INFO  [2013-09-30 15:47:55,551] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.2.9.59:3181, /10.169.0.213:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.145.202.79:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,552] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
>> ERROR [2013-09-30 15:47:55,552] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
>> WARN  [2013-09-30 15:47:55,552] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,552] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 2
>> ERROR [2013-09-30 15:47:55,552] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,557] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,558] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
>> ERROR [2013-09-30 15:47:55,560] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
>> WARN  [2013-09-30 15:47:55,560] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,560] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
>> ERROR [2013-09-30 15:47:55,560] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,560] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
>> INFO  [2013-09-30 15:47:55,562] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.2.9.59:3181, /10.72.43.204:3181], old meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.144.81.168:3181"
>>   ensembleMember: "10.72.43.204:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> , new meta data is
>> BookieMetadataFormatVersion	2
>> quorumSize: 2
>> ensembleSize: 3
>> length: 0
>> lastEntryId: -1
>> state: OPEN
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   ensembleMember: "10.232.13.238:3181"
>>   firstEntryId: 0
>> }
>> segment {
>>   ensembleMember: "10.206.11.160:3181"
>>   ensembleMember: "10.2.9.59:3181"
>>   ensembleMember: "10.169.0.213:3181"
>>   firstEntryId: 55
>> }
>> digestType: CRC32
>> password: "redacted"
>> ackQuorumSize: 2
>>
>> INFO  [2013-09-30 15:47:55,566] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,568] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
>> INFO  [2013-09-30 15:47:55,572] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
>> ERROR [2013-09-30 15:47:55,572] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
>> WARN  [2013-09-30 15:47:55,572] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,572] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 1
>> ERROR [2013-09-30 15:47:55,572] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>> INFO  [2013-09-30 15:47:55,572] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,573] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
>> INFO  [2013-09-30 15:47:55,576] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
>> ERROR [2013-09-30 15:47:55,576] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
>> WARN  [2013-09-30 15:47:55,576] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
>> INFO  [2013-09-30 15:47:55,576] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 2
>> ERROR [2013-09-30 15:47:55,576] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
>>
>>
>

Mime
View raw message