bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "FPJ" <fpjunque...@yahoo.com>
Subject RE: Synchronous send blocked forever by changing ensemble.
Date Wed, 02 Oct 2013 11:27:43 GMT
A timeout doesn´t sound right to me. If a client is not being able to make
progress, then the behavior I expect is that the app is notified that the
ledger has gone bad and no further operations are allowed. For synchronous
calls, we throw an exception and for asynchronous calls we return an error
code. 

 

Now I believe we have this in place already, so we simply need to get rid of
this kind of livelock.

 

-Flavio 

 

From: Sijie Guo [mailto:guosijie@gmail.com] 
Sent: 02 October 2013 00:24
To: bookkeeper-user@zookeeper.apache.org
Subject: Re: Synchronous send blocked forever by changing ensemble.

 

 

 

On Tue, Oct 1, 2013 at 1:51 PM, James Kebinger <jkebinger@hubspot.com
<mailto:jkebinger@hubspot.com> > wrote:

We're doing synchronous writes to BK using the addEntry method on
LedgerHandle. There's no way to set a timeout on this call, so we're having
threads blocked "forever" (we have to bounce API nodes to clear them) while
the bookkeeper client seems to spin its wheels connecting to various bk
servers and changing the ensemble. I'll paste some logs below. More
available upon request :)

 

Two questions:

1) Should the synchronous method have an optional timeout so we can take
measures to instantiate a new bookkeeper client?

2) Given the logs below, can anyone figure out why the reconnects didn't
eventually find a live BK? During this time other services were writing
successfully to the same book keeper cluster.

 

 

Hmm. It looks like that you are hitting the race condition : BOOKKEEPER-620
<https://issues.apache.org/jira/browse/BOOKKEEPER-620>  . The newly
connected connection is disconnected by previous disconnect event, which
generate disconnect event and disconnect following new connections. So the
ledger ends up changing bookies.

We are going to release 4.2.2 including this fix recently (properly this
week). If it is urgent, you could properly apply the patch in
BOOKKKEEPER-620 <https://issues.apache.org/jira/browse/BOOKKEEPER-620>
first.

For timeout, we could consider adding timeout for synchronous methods. If
you'd like, you could make a change for it. Or, I could create a jira for
this work.

-Sijie

 


 

 

We're on bookkeeper 4.2.1

 

INFO  [2013-09-30 15:47:55,172]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,172]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,172] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,172] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
2
ERROR [2013-09-30 15:47:55,172]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,175]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,177]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,181]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
ERROR [2013-09-30 15:47:55,181]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
<http://10.145.202.79:3181> 
WARN  [2013-09-30 15:47:55,181] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,181] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.145.202.79:3181 <http://10.145.202.79:3181>
index: 1
ERROR [2013-09-30 15:47:55,181]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,181] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.145.202.79:3181
<http://10.145.202.79:3181> , /10.72.43.204:3181 <http://10.72.43.204:3181>
], old meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  ensembleMember: "10.72.43.204:3181 <http://10.72.43.204:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: ""
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.145.202.79:3181 <http://10.145.202.79:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: ""
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,187]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,188]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,190]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,191]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,191]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,191] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,191] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
1
ERROR [2013-09-30 15:47:55,191]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,191]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,194]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,194]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
<http://10.72.43.204:3181> 
WARN  [2013-09-30 15:47:55,194] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,194] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.72.43.204:3181 <http://10.72.43.204:3181>
index: 2
ERROR [2013-09-30 15:47:55,194]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,200]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,202]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,205]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
ERROR [2013-09-30 15:47:55,205]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
<http://10.145.202.79:3181> 
WARN  [2013-09-30 15:47:55,205] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,205] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.145.202.79:3181 <http://10.145.202.79:3181>
index: 1
ERROR [2013-09-30 15:47:55,205]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,206] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.145.202.79:3181
<http://10.145.202.79:3181> , /10.2.9.59:3181 <http://10.2.9.59:3181> ], old
meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.145.202.79:3181 <http://10.145.202.79:3181> "
  ensembleMember: "10.72.43.204:3181 <http://10.72.43.204:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,211]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,212]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,215]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
ERROR [2013-09-30 15:47:55,216]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
<http://10.169.0.213:3181> 
WARN  [2013-09-30 15:47:55,216] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,216] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.169.0.213:3181 <http://10.169.0.213:3181>
index: 1
ERROR [2013-09-30 15:47:55,216]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,216]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,217]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,219]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,219]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,219] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,219] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
2
ERROR [2013-09-30 15:47:55,219]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,222]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,223]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,227]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
ERROR [2013-09-30 15:47:55,227]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
<http://10.145.202.79:3181> 
WARN  [2013-09-30 15:47:55,227] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,227] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.145.202.79:3181 index: 1
ERROR [2013-09-30 15:47:55,227]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,231] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.145.202.79:3181
<http://10.145.202.79:3181> , /10.232.13.238:3181
<http://10.232.13.238:3181> ], old meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.72.43.204:3181 <http://10.72.43.204:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.145.202.79:3181 <http://10.145.202.79:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,233]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,234]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,237]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,237]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
<http://10.72.43.204:3181> 
WARN  [2013-09-30 15:47:55,237] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,237] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.72.43.204:3181 <http://10.72.43.204:3181>
index: 1
ERROR [2013-09-30 15:47:55,237]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,240]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.232.13.238:3181 <http://10.232.13.238:3181> 
INFO  [2013-09-30 15:47:55,241]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.232.13.238:3181 <http://10.232.13.238:3181> 
INFO  [2013-09-30 15:47:55,243]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,243]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.232.13.238:3181 <http://10.232.13.238:3181> 
ERROR [2013-09-30 15:47:55,243]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
<http://10.232.13.238:3181> 
WARN  [2013-09-30 15:47:55,243] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,243] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.232.13.238:3181 <http://10.232.13.238:3181>
index: 2
ERROR [2013-09-30 15:47:55,243]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,244]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,246]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,246] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,246] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
1
ERROR [2013-09-30 15:47:55,247]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,247]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,250]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,251]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
ERROR [2013-09-30 15:47:55,254]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
<http://10.169.0.213:3181> 
WARN  [2013-09-30 15:47:55,254] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,254] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.169.0.213:3181 <http://10.169.0.213:3181>
index: 2
ERROR [2013-09-30 15:47:55,254]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,255]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,260] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.232.13.238:3181
<http://10.232.13.238:3181> , /10.169.0.213:3181 <http://10.169.0.213:3181>
], old meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  ensembleMember: "10.144.81.168:3181 <http://10.144.81.168:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,261]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,263]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,266]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
ERROR [2013-09-30 15:47:55,267]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
<http://10.144.81.168:3181> 
WARN  [2013-09-30 15:47:55,267] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,267] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.144.81.168:3181 <http://10.144.81.168:3181>
index: 2
ERROR [2013-09-30 15:47:55,267]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,271]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.232.13.238:3181 <http://10.232.13.238:3181> 
INFO  [2013-09-30 15:47:55,272]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.232.13.238:3181 <http://10.232.13.238:3181> 
INFO  [2013-09-30 15:47:55,272]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,274]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,276]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.232.13.238:3181 <http://10.232.13.238:3181> 
ERROR [2013-09-30 15:47:55,276]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
<http://10.232.13.238:3181> 
WARN  [2013-09-30 15:47:55,276] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,276] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.232.13.238:3181 <http://10.232.13.238:3181>
index: 1
ERROR [2013-09-30 15:47:55,276]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
ERROR [2013-09-30 15:47:55,278]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
<http://10.145.202.79:3181> 
WARN  [2013-09-30 15:47:55,278] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,278] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.145.202.79:3181 <http://10.145.202.79:3181>
index: 2
INFO  [2013-09-30 15:47:55,278]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
ERROR [2013-09-30 15:47:55,278]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,281]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,282]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,286]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,286] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,286] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
1
INFO  [2013-09-30 15:47:55,286]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,286]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,289] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.2.9.59:3181
<http://10.2.9.59:3181> , /10.72.43.204:3181 <http://10.72.43.204:3181> ],
old meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.144.81.168:3181 <http://10.144.81.168:3181> "
  ensembleMember: "10.72.43.204:3181 <http://10.72.43.204:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  ensembleMember: "10.145.202.79:3181 <http://10.145.202.79:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,318]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,320]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
ERROR [2013-09-30 15:47:55,324]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
<http://10.144.81.168:3181> 
WARN  [2013-09-30 15:47:55,324] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,324] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.144.81.168:3181 <http://10.144.81.168:3181>
index: 1
INFO  [2013-09-30 15:47:55,324]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
ERROR [2013-09-30 15:47:55,324]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,326]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,326]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,329]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,329]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
<http://10.72.43.204:3181> 
WARN  [2013-09-30 15:47:55,329] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,329] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.72.43.204:3181 <http://10.72.43.204:3181>
index: 2
ERROR [2013-09-30 15:47:55,329]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,330]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,332]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
ERROR [2013-09-30 15:47:55,335]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
<http://10.145.202.79:3181> 
WARN  [2013-09-30 15:47:55,335] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,335] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.145.202.79:3181 <http://10.145.202.79:3181>
index: 1
INFO  [2013-09-30 15:47:55,335]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
ERROR [2013-09-30 15:47:55,335]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,342]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,342]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,342]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,344]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,345]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,345]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
<http://10.72.43.204:3181> 
WARN  [2013-09-30 15:47:55,346] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,346] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.72.43.204:3181 <http://10.72.43.204:3181>
index: 1
ERROR [2013-09-30 15:47:55,346]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
ERROR [2013-09-30 15:47:55,348]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
<http://10.144.81.168:3181> 
WARN  [2013-09-30 15:47:55,348] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,348] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.144.81.168:3181 <http://10.144.81.168:3181>
index: 2
INFO  [2013-09-30 15:47:55,348]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
ERROR [2013-09-30 15:47:55,348]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,352]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,353]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,355] com.hubspot.connect.notify.SentryClient: Got
response code from Sentry notify: 429 - UNKNOWN STATUS CODE
INFO  [2013-09-30 15:47:55,355] com.hubspot.connect.notify.SentryClient:
Sentry responded to a 124722 character msg in 253 ms unsuccessfully
INFO  [2013-09-30 15:47:55,357]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,357]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,357] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,357] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
1
ERROR [2013-09-30 15:47:55,357]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,358] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.2.9.59:3181
<http://10.2.9.59:3181> , /10.232.13.238:3181 <http://10.232.13.238:3181> ],
old meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.145.202.79:3181 <http://10.145.202.79:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  ensembleMember: "10.144.81.168:3181 <http://10.144.81.168:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,362]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,364]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,368]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
ERROR [2013-09-30 15:47:55,368]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
<http://10.145.202.79:3181> 
WARN  [2013-09-30 15:47:55,368] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,368] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.145.202.79:3181 <http://10.145.202.79:3181>
index: 1
ERROR [2013-09-30 15:47:55,369]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,369]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.232.13.238:3181 <http://10.232.13.238:3181> 
INFO  [2013-09-30 15:47:55,370]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.232.13.238:3181 <http://10.232.13.238:3181> 
INFO  [2013-09-30 15:47:55,373]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.232.13.238:3181 <http://10.232.13.238:3181> 
ERROR [2013-09-30 15:47:55,374]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
<http://10.232.13.238:3181> 
WARN  [2013-09-30 15:47:55,374] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,374] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.232.13.238:3181 <http://10.232.13.238:3181>
index: 2
ERROR [2013-09-30 15:47:55,374]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,375]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,375]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,378]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,378]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
<http://10.72.43.204:3181> 
WARN  [2013-09-30 15:47:55,378] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,378] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.72.43.204:3181 <http://10.72.43.204:3181>
index: 1
ERROR [2013-09-30 15:47:55,378]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,384]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,385]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,385]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,387]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,388]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,388]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,388] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,388] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
1
ERROR [2013-09-30 15:47:55,389]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
ERROR [2013-09-30 15:47:55,390]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
<http://10.145.202.79:3181> 
WARN  [2013-09-30 15:47:55,390] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,390]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,390] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.145.202.79:3181 <http://10.145.202.79:3181>
index: 2
ERROR [2013-09-30 15:47:55,390]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,394]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,395]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,397]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,397]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
<http://10.72.43.204:3181> 
WARN  [2013-09-30 15:47:55,397] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,398] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.72.43.204:3181 <http://10.72.43.204:3181>
index: 1
ERROR [2013-09-30 15:47:55,398]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,399] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.72.43.204:3181
<http://10.72.43.204:3181> , /10.144.81.168:3181 <http://10.144.81.168:3181>
], old meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.144.81.168:3181 <http://10.144.81.168:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.72.43.204:3181 <http://10.72.43.204:3181> "
  ensembleMember: "10.145.202.79:3181 <http://10.145.202.79:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,404]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,405]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
ERROR [2013-09-30 15:47:55,408]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
<http://10.169.0.213:3181> 
WARN  [2013-09-30 15:47:55,408] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,408] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.169.0.213:3181 <http://10.169.0.213:3181>
index: 1
INFO  [2013-09-30 15:47:55,408]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
ERROR [2013-09-30 15:47:55,408]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,411]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,413]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,413]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,414]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,417]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
<http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,417]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
WARN  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.144.81.168:3181 <http://10.144.81.168:3181>
index: 2
INFO  [2013-09-30 15:47:55,417]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,417]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
ERROR [2013-09-30 15:47:55,417]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
<http://10.72.43.204:3181> 
WARN  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,417] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.72.43.204:3181 <http://10.72.43.204:3181>
index: 1
ERROR [2013-09-30 15:47:55,418]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,423]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,424]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,427] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.232.13.238:3181
<http://10.232.13.238:3181> , /10.2.9.59:3181 <http://10.2.9.59:3181> ], old
meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.72.43.204:3181 <http://10.72.43.204:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,427]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,427]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,427] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,427] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
2
ERROR [2013-09-30 15:47:55,428]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,434]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.232.13.238:3181 <http://10.232.13.238:3181> 
INFO  [2013-09-30 15:47:55,435]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.232.13.238:3181 <http://10.232.13.238:3181> 
ERROR [2013-09-30 15:47:55,438]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
<http://10.232.13.238:3181> 
WARN  [2013-09-30 15:47:55,438] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,438] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.232.13.238:3181 <http://10.232.13.238:3181>
index: 1
INFO  [2013-09-30 15:47:55,438]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.232.13.238:3181 <http://10.232.13.238:3181> 
ERROR [2013-09-30 15:47:55,438]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,440] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.232.13.238:3181
<http://10.232.13.238:3181> , /10.169.0.213:3181 <http://10.169.0.213:3181>
], old meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.72.43.204:3181 <http://10.72.43.204:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,444]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,445]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,448]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,448]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
<http://10.72.43.204:3181> 
WARN  [2013-09-30 15:47:55,448] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,448] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.72.43.204:3181 <http://10.72.43.204:3181>
index: 1
ERROR [2013-09-30 15:47:55,448]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,454]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,455]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,455]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,457]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,458]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,458]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,458] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,458] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
1
ERROR [2013-09-30 15:47:55,459]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,460]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
ERROR [2013-09-30 15:47:55,460]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
<http://10.169.0.213:3181> 
WARN  [2013-09-30 15:47:55,460] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,460] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.169.0.213:3181 <http://10.169.0.213:3181>
index: 2
ERROR [2013-09-30 15:47:55,460]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,464]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,466]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,469] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.144.81.168:3181
<http://10.144.81.168:3181> , /10.2.9.59:3181 <http://10.2.9.59:3181> ], old
meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.144.81.168:3181 <http://10.144.81.168:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.144.81.168:3181 <http://10.144.81.168:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
ERROR [2013-09-30 15:47:55,470]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
<http://10.144.81.168:3181> 
WARN  [2013-09-30 15:47:55,470] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,470] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.144.81.168:3181 <http://10.144.81.168:3181>
index: 1
INFO  [2013-09-30 15:47:55,470]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
ERROR [2013-09-30 15:47:55,470]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,474]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,475]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,477]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,477]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,477] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,477] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
2
ERROR [2013-09-30 15:47:55,477]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,478] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.145.202.79:3181
<http://10.145.202.79:3181> , /10.2.9.59:3181 <http://10.2.9.59:3181> ], old
meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.145.202.79:3181 <http://10.145.202.79:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.144.81.168:3181 <http://10.144.81.168:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,483]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,484]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,486]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,486]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
ERROR [2013-09-30 15:47:55,486]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
<http://10.169.0.213:3181> 
WARN  [2013-09-30 15:47:55,486] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,486] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.169.0.213:3181 <http://10.169.0.213:3181>
index: 2
ERROR [2013-09-30 15:47:55,486]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,488]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,492]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
ERROR [2013-09-30 15:47:55,492]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
<http://10.145.202.79:3181> 
WARN  [2013-09-30 15:47:55,492] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,492] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.145.202.79:3181 <http://10.145.202.79:3181>
index: 1
ERROR [2013-09-30 15:47:55,492]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,492]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.232.13.238:3181 <http://10.232.13.238:3181> 
INFO  [2013-09-30 15:47:55,493]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.232.13.238:3181 <http://10.232.13.238:3181> 
ERROR [2013-09-30 15:47:55,496]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.232.13.238:3181
<http://10.232.13.238:3181> 
WARN  [2013-09-30 15:47:55,496] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,496] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.232.13.238:3181 <http://10.232.13.238:3181>
index: 2
INFO  [2013-09-30 15:47:55,496]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.232.13.238:3181 <http://10.232.13.238:3181> 
ERROR [2013-09-30 15:47:55,496]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,502]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,503]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,503]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,505]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,507]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
ERROR [2013-09-30 15:47:55,507]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
<http://10.145.202.79:3181> 
WARN  [2013-09-30 15:47:55,508] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,508] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.145.202.79:3181 <http://10.145.202.79:3181>
index: 2
ERROR [2013-09-30 15:47:55,508]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,508]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,510]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,510] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,510] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
1
ERROR [2013-09-30 15:47:55,511]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,513]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,514]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,519]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
ERROR [2013-09-30 15:47:55,519]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
<http://10.169.0.213:3181> 
WARN  [2013-09-30 15:47:55,519] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,519] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.169.0.213:3181 <http://10.169.0.213:3181>
index: 2
ERROR [2013-09-30 15:47:55,519]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,524] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.145.202.79:3181
<http://10.145.202.79:3181> , /10.169.0.213:3181 <http://10.169.0.213:3181>
], old meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.145.202.79:3181 <http://10.145.202.79:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,527]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,527]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,530]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,530]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,530] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,530] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
2
ERROR [2013-09-30 15:47:55,530]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,536]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,536]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,537]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,538]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
INFO  [2013-09-30 15:47:55,539]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,539]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
<http://10.72.43.204:3181> 
WARN  [2013-09-30 15:47:55,539] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,539] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.72.43.204:3181 <http://10.72.43.204:3181>
index: 2
ERROR [2013-09-30 15:47:55,539]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,542]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181 <http://10.145.202.79:3181> 
ERROR [2013-09-30 15:47:55,542]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.145.202.79:3181
<http://10.145.202.79:3181> 
WARN  [2013-09-30 15:47:55,542] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,543] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.145.202.79:3181 <http://10.145.202.79:3181>
index: 1
ERROR [2013-09-30 15:47:55,543]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,546]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,548]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
INFO  [2013-09-30 15:47:55,551] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.2.9.59:3181
<http://10.2.9.59:3181> , /10.169.0.213:3181 <http://10.169.0.213:3181> ],
old meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.145.202.79:3181 <http://10.145.202.79:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,552]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181 <http://10.169.0.213:3181> 
ERROR [2013-09-30 15:47:55,552]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.169.0.213:3181
<http://10.169.0.213:3181> 
WARN  [2013-09-30 15:47:55,552] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,552] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.169.0.213:3181 <http://10.169.0.213:3181>
index: 2
ERROR [2013-09-30 15:47:55,552]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,557]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,558]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
ERROR [2013-09-30 15:47:55,560]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.2.9.59:3181
<http://10.2.9.59:3181> 
WARN  [2013-09-30 15:47:55,560] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,560] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.2.9.59:3181 <http://10.2.9.59:3181>  index:
1
ERROR [2013-09-30 15:47:55,560]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,560]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181 <http://10.2.9.59:3181> 
INFO  [2013-09-30 15:47:55,562] org.apache.bookkeeper.client.LedgerHandle:
Resolve ledger metadata conflict while changing ensemble to:
[/10.206.11.160:3181 <http://10.206.11.160:3181> , /10.2.9.59:3181
<http://10.2.9.59:3181> , /10.72.43.204:3181 <http://10.72.43.204:3181> ],
old meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.144.81.168:3181 <http://10.144.81.168:3181> "
  ensembleMember: "10.72.43.204:3181 <http://10.72.43.204:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
, new meta data is 
BookieMetadataFormatVersion 2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  ensembleMember: "10.232.13.238:3181 <http://10.232.13.238:3181> "
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181 <http://10.206.11.160:3181> "
  ensembleMember: "10.2.9.59:3181 <http://10.2.9.59:3181> "
  ensembleMember: "10.169.0.213:3181 <http://10.169.0.213:3181> "
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2
 
INFO  [2013-09-30 15:47:55,566]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,568]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
INFO  [2013-09-30 15:47:55,572]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181 <http://10.144.81.168:3181> 
ERROR [2013-09-30 15:47:55,572]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.144.81.168:3181
<http://10.144.81.168:3181> 
WARN  [2013-09-30 15:47:55,572] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,572] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.144.81.168:3181 <http://10.144.81.168:3181>
index: 1
ERROR [2013-09-30 15:47:55,572]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55
INFO  [2013-09-30 15:47:55,572]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to bookie:
/10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,573]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected
to bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
INFO  [2013-09-30 15:47:55,576]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181 <http://10.72.43.204:3181> 
ERROR [2013-09-30 15:47:55,576]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write request
for adding entry: 55 ledger-id: 12799942 bookie: /10.72.43.204:3181
<http://10.72.43.204:3181> 
WARN  [2013-09-30 15:47:55,576] org.apache.bookkeeper.client.PendingAddOp:
Write did not succeed: 12799942, 55
INFO  [2013-09-30 15:47:55,576] org.apache.bookkeeper.client.LedgerHandle:
Handling failure of bookie: /10.72.43.204:3181 <http://10.72.43.204:3181>
index: 2
ERROR [2013-09-30 15:47:55,576]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback method:
55

 


Mime
View raw message