bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Kebinger <jkebin...@hubspot.com>
Subject Synchronous send blocked forever by changing ensemble.
Date Tue, 01 Oct 2013 20:51:52 GMT
We're doing synchronous writes to BK using the addEntry method on
LedgerHandle. There's no way to set a timeout on this call, so we're having
threads blocked "forever" (we have to bounce API nodes to clear them) while
the bookkeeper client seems to spin its wheels connecting to various bk
servers and changing the ensemble. I'll paste some logs below. More
available upon request :)

Two questions:
1) Should the synchronous method have an optional timeout so we can take
measures to instantiate a new bookkeeper client?
2) Given the logs below, can anyone figure out why the reconnects didn't
eventually find a live BK? During this time other services were writing
successfully to the same book keeper cluster.


We're on bookkeeper 4.2.1

INFO  [2013-09-30 15:47:55,172]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,172]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,172]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,172]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 2
ERROR [2013-09-30 15:47:55,172]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,175]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,177]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,181]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181
ERROR [2013-09-30 15:47:55,181]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.145.202.79:3181
WARN  [2013-09-30 15:47:55,181]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,181]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.145.202.79:3181 index: 1
ERROR [2013-09-30 15:47:55,181]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,181]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.145.202.79:3181, /10.72.43.204:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.2.9.59:3181"
  ensembleMember: "10.72.43.204:3181"
  firstEntryId: 55
}
digestType: CRC32
password: ""
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.145.202.79:3181"
  ensembleMember: "10.2.9.59:3181"
  firstEntryId: 55
}
digestType: CRC32
password: ""
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,187]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,188]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,190]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,191]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,191]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,191]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,191]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 1
ERROR [2013-09-30 15:47:55,191]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,191]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,194]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,194]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.72.43.204:3181
WARN  [2013-09-30 15:47:55,194]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,194]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.72.43.204:3181 index: 2
ERROR [2013-09-30 15:47:55,194]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,200]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,202]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,205]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181
ERROR [2013-09-30 15:47:55,205]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.145.202.79:3181
WARN  [2013-09-30 15:47:55,205]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,205]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.145.202.79:3181 index: 1
ERROR [2013-09-30 15:47:55,205]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,206]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.145.202.79:3181, /10.2.9.59:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.2.9.59:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.145.202.79:3181"
  ensembleMember: "10.72.43.204:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,211]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,212]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,215]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181
ERROR [2013-09-30 15:47:55,216]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.169.0.213:3181
WARN  [2013-09-30 15:47:55,216]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,216]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.169.0.213:3181 index: 1
ERROR [2013-09-30 15:47:55,216]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,216]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,217]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,219]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,219]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,219]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,219]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 2
ERROR [2013-09-30 15:47:55,219]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,222]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,223]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,227]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181
ERROR [2013-09-30 15:47:55,227]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.145.202.79:3181
WARN  [2013-09-30 15:47:55,227]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,227]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.145.202.79:3181 index: 1
ERROR [2013-09-30 15:47:55,227]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,231]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.145.202.79:3181, /10.232.13.238:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.72.43.204:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.145.202.79:3181"
  ensembleMember: "10.2.9.59:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,233]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,234]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,237]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,237]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.72.43.204:3181
WARN  [2013-09-30 15:47:55,237]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,237]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.72.43.204:3181 index: 1
ERROR [2013-09-30 15:47:55,237]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,240]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.232.13.238:3181
INFO  [2013-09-30 15:47:55,241]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.232.13.238:3181
INFO  [2013-09-30 15:47:55,243]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,243]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.232.13.238:3181
ERROR [2013-09-30 15:47:55,243]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.232.13.238:3181
WARN  [2013-09-30 15:47:55,243]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,243]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.232.13.238:3181 index: 2
ERROR [2013-09-30 15:47:55,243]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,244]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,246]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,246]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,246]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 1
ERROR [2013-09-30 15:47:55,247]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,247]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,250]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,251]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.169.0.213:3181
ERROR [2013-09-30 15:47:55,254]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.169.0.213:3181
WARN  [2013-09-30 15:47:55,254]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,254]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.169.0.213:3181 index: 2
ERROR [2013-09-30 15:47:55,254]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,255]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,260]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.232.13.238:3181, /10.169.0.213:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.232.13.238:3181"
  ensembleMember: "10.144.81.168:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.2.9.59:3181"
  ensembleMember: "10.169.0.213:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,261]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,263]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,266]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181
ERROR [2013-09-30 15:47:55,267]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.144.81.168:3181
WARN  [2013-09-30 15:47:55,267]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,267]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.144.81.168:3181 index: 2
ERROR [2013-09-30 15:47:55,267]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,271]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.232.13.238:3181
INFO  [2013-09-30 15:47:55,272]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.232.13.238:3181
INFO  [2013-09-30 15:47:55,272]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,274]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,276]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.232.13.238:3181
ERROR [2013-09-30 15:47:55,276]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.232.13.238:3181
WARN  [2013-09-30 15:47:55,276]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,276]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.232.13.238:3181 index: 1
ERROR [2013-09-30 15:47:55,276]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
ERROR [2013-09-30 15:47:55,278]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.145.202.79:3181
WARN  [2013-09-30 15:47:55,278]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,278]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.145.202.79:3181 index: 2
INFO  [2013-09-30 15:47:55,278]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181
ERROR [2013-09-30 15:47:55,278]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,281]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,282]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,286]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,286]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,286]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 1
INFO  [2013-09-30 15:47:55,286]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,286]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,289]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.2.9.59:3181, /10.72.43.204:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.144.81.168:3181"
  ensembleMember: "10.72.43.204:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.2.9.59:3181"
  ensembleMember: "10.145.202.79:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,318]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,320]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.144.81.168:3181
ERROR [2013-09-30 15:47:55,324]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.144.81.168:3181
WARN  [2013-09-30 15:47:55,324]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,324]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.144.81.168:3181 index: 1
INFO  [2013-09-30 15:47:55,324]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181
ERROR [2013-09-30 15:47:55,324]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,326]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,326]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,329]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,329]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.72.43.204:3181
WARN  [2013-09-30 15:47:55,329]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,329]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.72.43.204:3181 index: 2
ERROR [2013-09-30 15:47:55,329]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,330]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,332]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.145.202.79:3181
ERROR [2013-09-30 15:47:55,335]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.145.202.79:3181
WARN  [2013-09-30 15:47:55,335]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,335]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.145.202.79:3181 index: 1
INFO  [2013-09-30 15:47:55,335]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181
ERROR [2013-09-30 15:47:55,335]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,342]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,342]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,342]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,344]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,345]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,345]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.72.43.204:3181
WARN  [2013-09-30 15:47:55,346]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,346]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.72.43.204:3181 index: 1
ERROR [2013-09-30 15:47:55,346]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
ERROR [2013-09-30 15:47:55,348]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.144.81.168:3181
WARN  [2013-09-30 15:47:55,348]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,348]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.144.81.168:3181 index: 2
INFO  [2013-09-30 15:47:55,348]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181
ERROR [2013-09-30 15:47:55,348]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,352]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,353]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
WARN  [2013-09-30 15:47:55,355]
com.hubspot.connect.notify.SentryClient: Got response code from Sentry
notify: 429 - UNKNOWN STATUS CODE
INFO  [2013-09-30 15:47:55,355]
com.hubspot.connect.notify.SentryClient: Sentry responded to a 124722
character msg in 253 ms unsuccessfully
INFO  [2013-09-30 15:47:55,357]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,357]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,357]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,357]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 1
ERROR [2013-09-30 15:47:55,357]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,358]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.2.9.59:3181, /10.232.13.238:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.145.202.79:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.2.9.59:3181"
  ensembleMember: "10.144.81.168:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,362]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,364]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,368]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181
ERROR [2013-09-30 15:47:55,368]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.145.202.79:3181
WARN  [2013-09-30 15:47:55,368]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,368]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.145.202.79:3181 index: 1
ERROR [2013-09-30 15:47:55,369]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,369]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.232.13.238:3181
INFO  [2013-09-30 15:47:55,370]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.232.13.238:3181
INFO  [2013-09-30 15:47:55,373]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.232.13.238:3181
ERROR [2013-09-30 15:47:55,374]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.232.13.238:3181
WARN  [2013-09-30 15:47:55,374]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,374]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.232.13.238:3181 index: 2
ERROR [2013-09-30 15:47:55,374]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,375]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,375]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,378]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,378]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.72.43.204:3181
WARN  [2013-09-30 15:47:55,378]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,378]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.72.43.204:3181 index: 1
ERROR [2013-09-30 15:47:55,378]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,384]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,385]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,385]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,387]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,388]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,388]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,388]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,388]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 1
ERROR [2013-09-30 15:47:55,389]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
ERROR [2013-09-30 15:47:55,390]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.145.202.79:3181
WARN  [2013-09-30 15:47:55,390]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,390]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,390]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.145.202.79:3181 index: 2
ERROR [2013-09-30 15:47:55,390]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,394]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,395]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,397]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,397]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.72.43.204:3181
WARN  [2013-09-30 15:47:55,397]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,398]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.72.43.204:3181 index: 1
ERROR [2013-09-30 15:47:55,398]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,399]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.72.43.204:3181, /10.144.81.168:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.144.81.168:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.72.43.204:3181"
  ensembleMember: "10.145.202.79:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,404]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,405]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.169.0.213:3181
ERROR [2013-09-30 15:47:55,408]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.169.0.213:3181
WARN  [2013-09-30 15:47:55,408]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,408]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.169.0.213:3181 index: 1
INFO  [2013-09-30 15:47:55,408]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181
ERROR [2013-09-30 15:47:55,408]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,411]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,413]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,413]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,414]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,417]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.144.81.168:3181
INFO  [2013-09-30 15:47:55,417]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181
WARN  [2013-09-30 15:47:55,417]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,417]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.144.81.168:3181 index: 2
INFO  [2013-09-30 15:47:55,417]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,417]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
ERROR [2013-09-30 15:47:55,417]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.72.43.204:3181
WARN  [2013-09-30 15:47:55,417]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,417]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.72.43.204:3181 index: 1
ERROR [2013-09-30 15:47:55,418]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,423]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,424]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,427]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.232.13.238:3181, /10.2.9.59:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.232.13.238:3181"
  ensembleMember: "10.2.9.59:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.72.43.204:3181"
  ensembleMember: "10.2.9.59:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,427]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,427]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,427]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,427]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 2
ERROR [2013-09-30 15:47:55,428]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,434]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.232.13.238:3181
INFO  [2013-09-30 15:47:55,435]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.232.13.238:3181
ERROR [2013-09-30 15:47:55,438]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.232.13.238:3181
WARN  [2013-09-30 15:47:55,438]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,438]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.232.13.238:3181 index: 1
INFO  [2013-09-30 15:47:55,438]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.232.13.238:3181
ERROR [2013-09-30 15:47:55,438]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,440]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.232.13.238:3181, /10.169.0.213:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.72.43.204:3181"
  ensembleMember: "10.169.0.213:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.232.13.238:3181"
  ensembleMember: "10.2.9.59:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,444]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,445]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,448]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,448]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.72.43.204:3181
WARN  [2013-09-30 15:47:55,448]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,448]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.72.43.204:3181 index: 1
ERROR [2013-09-30 15:47:55,448]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,454]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,455]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,455]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,457]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,458]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,458]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,458]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,458]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 1
ERROR [2013-09-30 15:47:55,459]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,460]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181
ERROR [2013-09-30 15:47:55,460]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.169.0.213:3181
WARN  [2013-09-30 15:47:55,460]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,460]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.169.0.213:3181 index: 2
ERROR [2013-09-30 15:47:55,460]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,464]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,466]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,469]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.144.81.168:3181, /10.2.9.59:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.144.81.168:3181"
  ensembleMember: "10.2.9.59:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.144.81.168:3181"
  ensembleMember: "10.169.0.213:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

ERROR [2013-09-30 15:47:55,470]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.144.81.168:3181
WARN  [2013-09-30 15:47:55,470]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,470]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.144.81.168:3181 index: 1
INFO  [2013-09-30 15:47:55,470]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181
ERROR [2013-09-30 15:47:55,470]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,474]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,475]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,477]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,477]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,477]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,477]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 2
ERROR [2013-09-30 15:47:55,477]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,478]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.145.202.79:3181, /10.2.9.59:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.145.202.79:3181"
  ensembleMember: "10.169.0.213:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.144.81.168:3181"
  ensembleMember: "10.2.9.59:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,483]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,484]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,486]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,486]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181
ERROR [2013-09-30 15:47:55,486]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.169.0.213:3181
WARN  [2013-09-30 15:47:55,486]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,486]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.169.0.213:3181 index: 2
ERROR [2013-09-30 15:47:55,486]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,488]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,492]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181
ERROR [2013-09-30 15:47:55,492]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.145.202.79:3181
WARN  [2013-09-30 15:47:55,492]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,492]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.145.202.79:3181 index: 1
ERROR [2013-09-30 15:47:55,492]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,492]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.232.13.238:3181
INFO  [2013-09-30 15:47:55,493]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.232.13.238:3181
ERROR [2013-09-30 15:47:55,496]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.232.13.238:3181
WARN  [2013-09-30 15:47:55,496]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,496]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.232.13.238:3181 index: 2
INFO  [2013-09-30 15:47:55,496]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.232.13.238:3181
ERROR [2013-09-30 15:47:55,496]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,502]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,503]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,503]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,505]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,507]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181
ERROR [2013-09-30 15:47:55,507]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.145.202.79:3181
WARN  [2013-09-30 15:47:55,508]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,508]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.145.202.79:3181 index: 2
ERROR [2013-09-30 15:47:55,508]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,508]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,510]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,510]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,510]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 1
ERROR [2013-09-30 15:47:55,511]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,513]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,514]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,519]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181
ERROR [2013-09-30 15:47:55,519]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.169.0.213:3181
WARN  [2013-09-30 15:47:55,519]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,519]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.169.0.213:3181 index: 2
ERROR [2013-09-30 15:47:55,519]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,524]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.145.202.79:3181, /10.169.0.213:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.145.202.79:3181"
  ensembleMember: "10.2.9.59:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.2.9.59:3181"
  ensembleMember: "10.169.0.213:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,527]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,527]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,530]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,530]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,530]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,530]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 2
ERROR [2013-09-30 15:47:55,530]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,536]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,536]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,537]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,538]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.145.202.79:3181
INFO  [2013-09-30 15:47:55,539]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,539]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.72.43.204:3181
WARN  [2013-09-30 15:47:55,539]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,539]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.72.43.204:3181 index: 2
ERROR [2013-09-30 15:47:55,539]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,542]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.145.202.79:3181
ERROR [2013-09-30 15:47:55,542]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.145.202.79:3181
WARN  [2013-09-30 15:47:55,542]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,543]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.145.202.79:3181 index: 1
ERROR [2013-09-30 15:47:55,543]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,546]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,548]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.169.0.213:3181
INFO  [2013-09-30 15:47:55,551]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.2.9.59:3181, /10.169.0.213:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.2.9.59:3181"
  ensembleMember: "10.169.0.213:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.145.202.79:3181"
  ensembleMember: "10.169.0.213:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,552]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.169.0.213:3181
ERROR [2013-09-30 15:47:55,552]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.169.0.213:3181
WARN  [2013-09-30 15:47:55,552]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,552]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.169.0.213:3181 index: 2
ERROR [2013-09-30 15:47:55,552]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,557]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,558]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.2.9.59:3181
ERROR [2013-09-30 15:47:55,560]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.2.9.59:3181
WARN  [2013-09-30 15:47:55,560]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,560]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.2.9.59:3181 index: 1
ERROR [2013-09-30 15:47:55,560]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,560]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.2.9.59:3181
INFO  [2013-09-30 15:47:55,562]
org.apache.bookkeeper.client.LedgerHandle: Resolve ledger metadata
conflict while changing ensemble to: [/10.206.11.160:3181,
/10.2.9.59:3181, /10.72.43.204:3181], old meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.144.81.168:3181"
  ensembleMember: "10.72.43.204:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

, new meta data is
BookieMetadataFormatVersion	2
quorumSize: 2
ensembleSize: 3
length: 0
lastEntryId: -1
state: OPEN
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.169.0.213:3181"
  ensembleMember: "10.232.13.238:3181"
  firstEntryId: 0
}
segment {
  ensembleMember: "10.206.11.160:3181"
  ensembleMember: "10.2.9.59:3181"
  ensembleMember: "10.169.0.213:3181"
  firstEntryId: 55
}
digestType: CRC32
password: "redacted"
ackQuorumSize: 2

INFO  [2013-09-30 15:47:55,566]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,568]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.144.81.168:3181
INFO  [2013-09-30 15:47:55,572]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.144.81.168:3181
ERROR [2013-09-30 15:47:55,572]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.144.81.168:3181
WARN  [2013-09-30 15:47:55,572]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,572]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.144.81.168:3181 index: 1
ERROR [2013-09-30 15:47:55,572]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55
INFO  [2013-09-30 15:47:55,572]
org.apache.bookkeeper.proto.PerChannelBookieClient: Connecting to
bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,573]
org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully
connected to bookie: /10.72.43.204:3181
INFO  [2013-09-30 15:47:55,576]
org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from
bookie: /10.72.43.204:3181
ERROR [2013-09-30 15:47:55,576]
org.apache.bookkeeper.proto.PerChannelBookieClient: Could not write
request for adding entry: 55 ledger-id: 12799942 bookie:
/10.72.43.204:3181
WARN  [2013-09-30 15:47:55,576]
org.apache.bookkeeper.client.PendingAddOp: Write did not succeed:
12799942, 55
INFO  [2013-09-30 15:47:55,576]
org.apache.bookkeeper.client.LedgerHandle: Handling failure of bookie:
/10.72.43.204:3181 index: 2
ERROR [2013-09-30 15:47:55,576]
org.apache.bookkeeper.proto.PerChannelBookieClient: Invoked callback
method: 55

Mime
View raw message