zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Schilli...@perlmeister.com>
Subject Re: Total Outage of all ZooKeeper Instances
Date Wed, 12 Oct 2011 15:46:38 GMT
On Wed, 12 Oct 2011, Ted Dunning wrote:

> Without a bit of log info, I don't think we would have any idea what
> the problem was other than the -7 return code.
> 
> Can you give some specifics about what the ZK instances were saying?

Sure, this is what I found in the zookeeper logs around the same time:

2011-10-06 21:14:32.322544500 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634]
- EndOfStreamExcepti
on: Unable to read additional data from client sessionid 0x3324ffac32a0012, likely client
has closed socket
2011-10-06 21:14:32.322860500 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435]
- Closed socket con
nection for client /127.0.0.1:44431 which had sessionid 0x3324ffac32a0012
2011-10-06 21:14:32.829464500 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634]
- EndOfStreamExcepti
on: Unable to read additional data from client sessionid 0x3324ffac32a0011, likely client
has closed socket
2011-10-06 21:14:32.829468500 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435]
- Closed socket con
nection for client /127.0.0.1:44435 which had sessionid 0x3324ffac32a0011

Followed by a number of client reconnect attempts:

2011-10-06 21:14:34.956557500 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@770]
- Client attempting 
to renew session 0x3324ffac32a000f at /127.0.0.1:45019
2011-10-06 21:14:34.956559500 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103]
- Revalidating client: 230
334469384044559
2011-10-06 21:14:34.957578500 INFO  [QuorumPeer:/0.0.0.0:2181:NIOServerCnxn@1580] - Established
session 0x3324ffac32a0
00f with negotiated timeout 10000 for client /127.0.0.1:45019
2011-10-06 21:14:35.199455500 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634]
- EndOfStreamExcepti
on: Unable to read additional data from client sessionid 0x3324ffac32a000e, likely client
has closed socket
2011-10-06 21:14:35.199457500 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435]
- Closed socket con
nection for client /127.0.0.1:44434 which had sessionid 0x3324ffac32a000e
2011-10-06 21:14:35.577628500 ERROR [CommitProcessor:3:NIOServerCnxn@445] - Unexpected Exception:

2011-10-06 21:14:35.577629500 java.nio.channels.CancelledKeyException
2011-10-06 21:14:35.577631500   at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
2011-10-06 21:14:35.577632500   at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
2011-10-06 21:14:35.577633500   at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
2011-10-06 21:14:35.577634500   at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
2011-10-06 21:14:35.577641500   at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProces
sor.java:367)
2011-10-06 21:14:35.577642500   at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
2011-10-06 21:14:35.577797500 ERROR [CommitProcessor:3:NIOServerCnxn@445] - Unexpected Exception:

2011-10-06 21:14:35.577798500 java.nio.channels.CancelledKeyException
2011-10-06 21:14:35.577800500   at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
2011-10-06 21:14:35.577801500   at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
2011-10-06 21:14:35.577802500   at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
2011-10-06 21:14:35.577804500   at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
2011-10-06 21:14:35.577830500   at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProces
sor.java:171)
2011-10-06 21:14:35.577831500   at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)


-- 
-- Mike

Mike Schilli
m@perlmeister.com

Mime
View raw message