zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepinder Singh Setia <dse...@juniper.net>
Subject OperationTimeoutException error
Date Fri, 09 Aug 2013 16:13:21 GMT
I am using Kazoo client and see the following error:

Aug  9 07:07:20 a2s1 python[2085]: File "/usr/lib/python2.7/site-packages/kazoo/client.py",
line 894, in set
Aug  9 07:07:20 a2s1 python[2085]: return self.set_async(path, data, version).get()
Aug  9 07:07:20 a2s1 python[2085]: File "/usr/lib64/python2.7/site-packages/gevent/event.py",
line 223, in get
Aug  9 07:07:20 a2s1 python[2085]: raise self._exception 
Aug  9 07:07:20 a2s1 python[2085]: OperationTimeoutException: operation timeout

zookeeper logs around the error time:

2013-08-09 07:07:06,580 [myid:] - WARN  [SyncThread:0:FileTxnLog@321] - fsync-ing the write
ahead log in SyncThread:0 took 2291ms which will adversely effect operation latency. See the
ZooKeeper troubleshooting guide
…
2013-08-09 07:07:19,827 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349]
- caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1406356c3bc0001,
likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
        at java.lang.Thread.run(Thread.java:722)
2013-08-09 07:07:19,828 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1000]
- Closed socket connection for client /127.0.0.1:54698 which had sessionid 0x1406356c3bc0001
2013-08-09 07:07:20,246 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349]
- caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1406356c3bc0000,
likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
        at java.lang.Thread.run(Thread.java:722)
2013-08-09 07:07:20,246 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1000]
- Closed socket connection for client /127.0.0.1:54579 which had sessionid 0x1406356c3bc0000
2013-08-09 07:07:21,583 [myid:] - WARN  [SyncThread:0:FileTxnLog@321] - fsync-ing the write
ahead log in SyncThread:0 took 8001ms which will adversely effect operation latency. See the
ZooKeeper troubleshooting guide
2013-08-09 07:07:21,680 [myid:] - ERROR [SyncThread:0:NIOServerCnxn@180] - Unexpected Exception:
java.nio.channels.CancelledKeyException

[root@a2s1 zookeeper]# grep fsync /var/log/zookeeper/zookeeper.out 
2013-08-09 07:06:52,280 [myid:] - WARN  [SyncThread:0:FileTxnLog@321] - fsync-ing the write
ahead log in SyncThread:0 took 1724ms which will adversely effect operation latency. See the
ZooKeeper troubleshooting guide
2013-08-09 07:06:58,315 [myid:] - WARN  [SyncThread:0:FileTxnLog@321] - fsync-ing the write
ahead log in SyncThread:0 took 2378ms which will adversely effect operation latency. See the
ZooKeeper troubleshooting guide
2013-08-09 07:07:01,389 [myid:] - WARN  [SyncThread:0:FileTxnLog@321] - fsync-ing the write
ahead log in SyncThread:0 took 1113ms which will adversely effect operation latency. See the
ZooKeeper troubleshooting guide
2013-08-09 07:07:06,580 [myid:] - WARN  [SyncThread:0:FileTxnLog@321] - fsync-ing the write
ahead log in SyncThread:0 took 2291ms which will adversely effect operation latency. See the
ZooKeeper troubleshooting guide
2013-08-09 07:07:21,583 [myid:] - WARN  [SyncThread:0:FileTxnLog@321] - fsync-ing the write
ahead log in SyncThread:0 took 8001ms which will adversely effect operation latency. See the
ZooKeeper troubleshooting guide
[

Could the client (Kazoo) be timing out because of fsync delay? What parameter would control
duration for OperationTimeoutException that I can perhaps increase to verify? There is only
ZooKeeper client and the load isn't much - 1 read/sec and 2 writes/sec roughly. Zookeeper
configuration is default. Kazoo client params are also default. 

Thanks
Deepinder



Mime
View raw message