bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Kelly <iv...@apache.org>
Subject Re: Synchronous send blocked forever by changing ensemble.
Date Wed, 02 Oct 2013 13:11:43 GMT
Hi James,

Do you know what size the entry being written was? Currently the
server rejects messages over 1MB as invalid, but there's no special
code to handle this on the client.

-Ivan


On Wed, Oct 02, 2013 at 08:25:34AM -0400, James Kebinger wrote:
> Thanks Sijie, we'll look forward to installing 4.2.2 . In the meantime
> we're replacing our synchronous call with an asynchronous call combined
> with a countdown latch and timeout so we can throw away the client instance
> if this happens again.
> 
> 
> On Tue, Oct 1, 2013 at 7:24 PM, Sijie Guo <guosijie@gmail.com> wrote:
> 
> >
> >
> >
> > On Tue, Oct 1, 2013 at 1:51 PM, James Kebinger <jkebinger@hubspot.com>wrote:
> >
> >> We're doing synchronous writes to BK using the addEntry method on
> >> LedgerHandle. There's no way to set a timeout on this call, so we're having
> >> threads blocked "forever" (we have to bounce API nodes to clear them) while
> >> the bookkeeper client seems to spin its wheels connecting to various bk
> >> servers and changing the ensemble. I'll paste some logs below. More
> >> available upon request :)
> >>
> >> Two questions:
> >> 1) Should the synchronous method have an optional timeout so we can take
> >> measures to instantiate a new bookkeeper client?
> >>
> >> 2) Given the logs below, can anyone figure out why the reconnects didn't
> >> eventually find a live BK? During this time other services were writing
> >> successfully to the same book keeper cluster.
> >>
> >>
> > Hmm. It looks like that you are hitting the race condition :
> > BOOKKEEPER-620 <https://issues.apache.org/jira/browse/BOOKKEEPER-620> .
> > The newly connected connection is disconnected by previous disconnect
> > event, which generate disconnect event and disconnect following new
> > connections. So the ledger ends up changing bookies.
> >
> > We are going to release 4.2.2 including this fix recently (properly this
> > week). If it is urgent, you could properly apply the patch in
> > BOOKKKEEPER-620 <https://issues.apache.org/jira/browse/BOOKKEEPER-620>first.
> >
> > For timeout, we could consider adding timeout for synchronous methods. If
> > you'd like, you could make a change for it. Or, I could create a jira for
> > this work.
> >
> > -Sijie
> >
> >
> >
> >
> >>
> >> We're on bookkeeper 4.2.1
> >>
> >> INFO  [2013-09-30 15:47:55,172] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,172] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,172] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,172] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,172] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,175] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,177] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,181] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
> >> ERROR [2013-09-30 15:47:55,181] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
> >> WARN  [2013-09-30 15:47:55,181] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,181] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,181] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,181] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.145.202.79:3181, /10.72.43.204:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   ensembleMember: "10.72.43.204:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: ""
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.145.202.79:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: ""
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,187] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,188] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,190] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,191] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,191] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,191] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,191] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,191] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,191] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,194] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,194] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
> >> WARN  [2013-09-30 15:47:55,194] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,194] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,194] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,200] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,202] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,205] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
> >> ERROR [2013-09-30 15:47:55,205] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
> >> WARN  [2013-09-30 15:47:55,205] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,205] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,205] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,206] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.145.202.79:3181, /10.2.9.59:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.145.202.79:3181"
> >>   ensembleMember: "10.72.43.204:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,211] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,212] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,215] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
> >> ERROR [2013-09-30 15:47:55,216] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
> >> WARN  [2013-09-30 15:47:55,216] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,216] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,216] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,216] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,217] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,219] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,219] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,219] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,219] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,219] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,222] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,223] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,227] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
> >> ERROR [2013-09-30 15:47:55,227] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
> >> WARN  [2013-09-30 15:47:55,227] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,227] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,227] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,231] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.145.202.79:3181, /10.232.13.238:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.72.43.204:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.145.202.79:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,233] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,234] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,237] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,237] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
> >> WARN  [2013-09-30 15:47:55,237] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,237] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,237] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,240] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.232.13.238:3181
> >> INFO  [2013-09-30 15:47:55,241] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.232.13.238:3181
> >> INFO  [2013-09-30 15:47:55,243] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,243] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.232.13.238:3181
> >> ERROR [2013-09-30 15:47:55,243] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
> >> WARN  [2013-09-30 15:47:55,243] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,243] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.232.13.238:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,243] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,244] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,246] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,246] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,246] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,247] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,247] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,250] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,251] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
> >> ERROR [2013-09-30 15:47:55,254] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
> >> WARN  [2013-09-30 15:47:55,254] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,254] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,254] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,255] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,260] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.232.13.238:3181, /10.169.0.213:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   ensembleMember: "10.144.81.168:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,261] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,263] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,266] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
> >> ERROR [2013-09-30 15:47:55,267] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
> >> WARN  [2013-09-30 15:47:55,267] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,267] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,267] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,271] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.232.13.238:3181
> >> INFO  [2013-09-30 15:47:55,272] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.232.13.238:3181
> >> INFO  [2013-09-30 15:47:55,272] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,274] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,276] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.232.13.238:3181
> >> ERROR [2013-09-30 15:47:55,276] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
> >> WARN  [2013-09-30 15:47:55,276] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,276] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.232.13.238:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,276] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> ERROR [2013-09-30 15:47:55,278] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
> >> WARN  [2013-09-30 15:47:55,278] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,278] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 2
> >> INFO  [2013-09-30 15:47:55,278] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
> >> ERROR [2013-09-30 15:47:55,278] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,281] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,282] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,286] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,286] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,286] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
> >> INFO  [2013-09-30 15:47:55,286] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,286] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,289] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.2.9.59:3181, /10.72.43.204:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.144.81.168:3181"
> >>   ensembleMember: "10.72.43.204:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   ensembleMember: "10.145.202.79:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,318] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,320] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
> >> ERROR [2013-09-30 15:47:55,324] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
> >> WARN  [2013-09-30 15:47:55,324] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,324] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 1
> >> INFO  [2013-09-30 15:47:55,324] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
> >> ERROR [2013-09-30 15:47:55,324] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,326] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,326] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,329] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,329] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
> >> WARN  [2013-09-30 15:47:55,329] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,329] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,329] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,330] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,332] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
> >> ERROR [2013-09-30 15:47:55,335] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
> >> WARN  [2013-09-30 15:47:55,335] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,335] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
> >> INFO  [2013-09-30 15:47:55,335] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
> >> ERROR [2013-09-30 15:47:55,335] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,342] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,342] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,342] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,344] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,345] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,345] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
> >> WARN  [2013-09-30 15:47:55,346] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,346] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,346] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> ERROR [2013-09-30 15:47:55,348] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
> >> WARN  [2013-09-30 15:47:55,348] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,348] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 2
> >> INFO  [2013-09-30 15:47:55,348] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
> >> ERROR [2013-09-30 15:47:55,348] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,352] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,353] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,355] com.hubspot.connect.notify.SentryClient: Got response code from Sentry notify: 429 - UNKNOWN STATUS CODE
> >> INFO  [2013-09-30 15:47:55,355] com.hubspot.connect.notify.SentryClient: Sentry responded to a 124722 character msg in 253 ms unsuccessfully
> >> INFO  [2013-09-30 15:47:55,357] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,357] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,357] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,357] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,357] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,358] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.2.9.59:3181, /10.232.13.238:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.145.202.79:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   ensembleMember: "10.144.81.168:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,362] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,364] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,368] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
> >> ERROR [2013-09-30 15:47:55,368] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
> >> WARN  [2013-09-30 15:47:55,368] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,368] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,369] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,369] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.232.13.238:3181
> >> INFO  [2013-09-30 15:47:55,370] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.232.13.238:3181
> >> INFO  [2013-09-30 15:47:55,373] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.232.13.238:3181
> >> ERROR [2013-09-30 15:47:55,374] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
> >> WARN  [2013-09-30 15:47:55,374] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,374] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.232.13.238:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,374] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,375] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,375] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,378] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,378] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
> >> WARN  [2013-09-30 15:47:55,378] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,378] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,378] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,384] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,385] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,385] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,387] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,388] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,388] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,388] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,388] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,389] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> ERROR [2013-09-30 15:47:55,390] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
> >> WARN  [2013-09-30 15:47:55,390] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,390] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,390] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,390] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,394] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,395] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,397] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,397] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
> >> WARN  [2013-09-30 15:47:55,397] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,398] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,398] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,399] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.72.43.204:3181, /10.144.81.168:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.144.81.168:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.72.43.204:3181"
> >>   ensembleMember: "10.145.202.79:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,404] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,405] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
> >> ERROR [2013-09-30 15:47:55,408] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
> >> WARN  [2013-09-30 15:47:55,408] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,408] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 1
> >> INFO  [2013-09-30 15:47:55,408] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
> >> ERROR [2013-09-30 15:47:55,408] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,411] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,413] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,413] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,414] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,417] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,417] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
> >> WARN  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 2
> >> INFO  [2013-09-30 15:47:55,417] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,417] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> ERROR [2013-09-30 15:47:55,417] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
> >> WARN  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,418] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,423] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,424] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,427] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.232.13.238:3181, /10.2.9.59:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.72.43.204:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,427] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,427] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,427] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,427] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,428] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,434] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.232.13.238:3181
> >> INFO  [2013-09-30 15:47:55,435] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.232.13.238:3181
> >> ERROR [2013-09-30 15:47:55,438] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
> >> WARN  [2013-09-30 15:47:55,438] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,438] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.232.13.238:3181 index: 1
> >> INFO  [2013-09-30 15:47:55,438] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.232.13.238:3181
> >> ERROR [2013-09-30 15:47:55,438] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,440] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.232.13.238:3181, /10.169.0.213:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.72.43.204:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,444] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,445] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,448] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,448] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
> >> WARN  [2013-09-30 15:47:55,448] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,448] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,448] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,454] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,455] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,455] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,457] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,458] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,458] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,458] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,458] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,459] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,460] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
> >> ERROR [2013-09-30 15:47:55,460] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
> >> WARN  [2013-09-30 15:47:55,460] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,460] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,460] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,464] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,466] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,469] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.144.81.168:3181, /10.2.9.59:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.144.81.168:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.144.81.168:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> ERROR [2013-09-30 15:47:55,470] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
> >> WARN  [2013-09-30 15:47:55,470] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,470] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 1
> >> INFO  [2013-09-30 15:47:55,470] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
> >> ERROR [2013-09-30 15:47:55,470] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,474] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,475] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,477] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,477] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,477] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,477] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,477] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,478] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.145.202.79:3181, /10.2.9.59:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.145.202.79:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.144.81.168:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,483] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,484] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,486] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,486] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
> >> ERROR [2013-09-30 15:47:55,486] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
> >> WARN  [2013-09-30 15:47:55,486] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,486] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,486] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,488] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,492] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
> >> ERROR [2013-09-30 15:47:55,492] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
> >> WARN  [2013-09-30 15:47:55,492] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,492] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,492] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,492] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.232.13.238:3181
> >> INFO  [2013-09-30 15:47:55,493] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.232.13.238:3181
> >> ERROR [2013-09-30 15:47:55,496] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
> >> WARN  [2013-09-30 15:47:55,496] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,496] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.232.13.238:3181 index: 2
> >> INFO  [2013-09-30 15:47:55,496] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.232.13.238:3181
> >> ERROR [2013-09-30 15:47:55,496] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,502] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,503] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,503] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,505] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,507] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
> >> ERROR [2013-09-30 15:47:55,507] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
> >> WARN  [2013-09-30 15:47:55,508] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,508] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,508] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,508] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,510] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,510] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,510] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,511] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,513] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,514] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,519] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
> >> ERROR [2013-09-30 15:47:55,519] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
> >> WARN  [2013-09-30 15:47:55,519] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,519] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,519] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,524] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.145.202.79:3181, /10.169.0.213:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.145.202.79:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,527] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,527] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,530] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,530] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,530] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,530] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,530] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,536] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,536] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,537] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,538] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.145.202.79:3181
> >> INFO  [2013-09-30 15:47:55,539] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,539] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
> >> WARN  [2013-09-30 15:47:55,539] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,539] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,539] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,542] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.145.202.79:3181
> >> ERROR [2013-09-30 15:47:55,542] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
> >> WARN  [2013-09-30 15:47:55,542] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,543] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.145.202.79:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,543] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,546] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,548] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.169.0.213:3181
> >> INFO  [2013-09-30 15:47:55,551] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.2.9.59:3181, /10.169.0.213:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.145.202.79:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,552] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.169.0.213:3181
> >> ERROR [2013-09-30 15:47:55,552] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
> >> WARN  [2013-09-30 15:47:55,552] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,552] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.169.0.213:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,552] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,557] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,558] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.2.9.59:3181
> >> ERROR [2013-09-30 15:47:55,560] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
> >> WARN  [2013-09-30 15:47:55,560] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,560] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.2.9.59:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,560] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,560] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.2.9.59:3181
> >> INFO  [2013-09-30 15:47:55,562] org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata conflict while changing ensemble to: [/10.206.11.160:3181, /10.2.9.59:3181, /10.72.43.204:3181], old meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.144.81.168:3181"
> >>   ensembleMember: "10.72.43.204:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> , new meta data is
> >> BookieMetadataFormatVersion	2
> >> quorumSize: 2
> >> ensembleSize: 3
> >> length: 0
> >> lastEntryId: -1
> >> state: OPEN
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   ensembleMember: "10.232.13.238:3181"
> >>   firstEntryId: 0
> >> }
> >> segment {
> >>   ensembleMember: "10.206.11.160:3181"
> >>   ensembleMember: "10.2.9.59:3181"
> >>   ensembleMember: "10.169.0.213:3181"
> >>   firstEntryId: 55
> >> }
> >> digestType: CRC32
> >> password: "redacted"
> >> ackQuorumSize: 2
> >>
> >> INFO  [2013-09-30 15:47:55,566] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,568] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.144.81.168:3181
> >> INFO  [2013-09-30 15:47:55,572] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.144.81.168:3181
> >> ERROR [2013-09-30 15:47:55,572] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
> >> WARN  [2013-09-30 15:47:55,572] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,572] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.144.81.168:3181 index: 1
> >> ERROR [2013-09-30 15:47:55,572] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >> INFO  [2013-09-30 15:47:55,572] org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,573] org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to bookie: /10.72.43.204:3181
> >> INFO  [2013-09-30 15:47:55,576] org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie: /10.72.43.204:3181
> >> ERROR [2013-09-30 15:47:55,576] org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
> >> WARN  [2013-09-30 15:47:55,576] org.apache.bookkeeper.client.PendingAddOp: Write did not succeed: 12799942, 55
> >> INFO  [2013-09-30 15:47:55,576] org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie: /10.72.43.204:3181 index: 2
> >> ERROR [2013-09-30 15:47:55,576] org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method: 55
> >>
> >>
> >

Mime
View raw message