hadoop-zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mahadev konar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (ZOOKEEPER-344) doIO in NioServerCnxn: Exception causing close of session : cause is "read error"
Date Wed, 01 Apr 2009 18:18:13 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694660#action_12694660
] 

Mahadev konar commented on ZOOKEEPER-344:
-----------------------------------------

for the first trace, closed != expired is true. it seems like the client reconnected to the
server and didnt expire in the first case.

in the second case it did expire. 

You are on a windows machine right? and also its a single server ensemble? I dont usually
use a windows box but I did notice once  that creating a log file with size of 64MB initially
took 
a lot of time (5-15 seconds and was repeatable) on my windows box with no other load . Just
to give you background, zookeeper takes snaphots and has transaction logs. The transaction
logs 
are created with an initialize size of 64MB so that we do not have to update the size of the
file repeatedly. This operation did take a lot of time on my windows machine and this did
cause huge
latencies in my zookeeper clients and lots of timeouts and session expired. 

Ill take another look at your logs to see if i can find something that might explain the frequent
session timeouts.

> doIO in NioServerCnxn: Exception causing close of session : cause is "read error"
> ---------------------------------------------------------------------------------
>
>                 Key: ZOOKEEPER-344
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-344
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: java client, server
>    Affects Versions: 3.1.0
>         Environment: jdk1.6.0_07
> Linux blade2 2.6.27.7-134.fc10.x86_64 #1 SMP Mon Dec 1 22:21:35 EST 2008 x86_64 x86_64
x86_64 GNU/Linux
>            Reporter: bryan thompson
>             Fix For: 3.2.0
>
>
> I have been having a problem with zookeeper 3.0.1 and now with 3.1.0 where I see a lot
of expired sessions.  I am using a 16 node cluster which is all on the same local network.
 There is a single zookeeper instance (these are benchmarking runs).
> The problem appears to be correlated with either run time or system load.\
> Personally I think that it is system load because I have session session expired events
under a Windows platform running zookeeper and the application (i.e., everthing is local)
when the application load suddenly spikes.  To me this suggests that the client is not able
to renew (ping) the zookeeper service in a timely manner and is expired.  But the log messages
below with the "read error" suggest that maybe there is something else going on?
> Zookeeper Configuration
> #Wed Mar 18 12:41:05 GMT-05:00 2009
> clientPort=2181
> dataDir=/var/bigdata/benchmark/zookeeper/1
> syncLimit=2
> dataLogDir=/var/bigdata/benchmark/zookeeper/1
> tickTime=2000
> Some representative log messages are below.
> Client side messages (from our app)
> ERROR [main-EventThread] com.bigdata.zookeeper.ZLockImpl$ZLockWatcher.process(ZLockImpl.java:400)
2009-03-18 13:35:40,335 - Session expired: WatchedEvent: Server state change. New state: Expired
: zpath=/benchmark/jobs/com.bigdata.service.jini.benchmark.ThroughputMaster/test_1/client1160/locknode
> ERROR [main-EventThread] com.bigdata.zookeeper.ZLockImpl$ZLockWatcher.process(ZLockImpl.java:400)
2009-03-18 13:35:40,335 - Session expired: WatchedEvent: Server state change. New state: Expired
: zpath=/benchmark/jobs/com.bigdata.service.jini.benchmark.ThroughputMaster/test_1/client1356/locknode
> Server side messages:
>  WARN [NIOServerCxn.Factory:2181] org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:417)
2009-03-18 13:06:57,252 - Exception causing close of session 0x1201aac14300022 due to java.io.IOException:
Read error
>  WARN [NIOServerCxn.Factory:2181] org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:417)
2009-03-18 13:06:58,198 - Exception causing close of session 0x1201aac1430000f due to java.io.IOException:
Read error

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message