zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yeshuangshuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-3211) zookeeper standalone mode,found a high level bug in kernel of centos7.0 ,zookeeper Server's tcp/ip socket connections(default 60 ) are CLOSE_WAIT ,this lead to zk can't work for client any more
Date Tue, 12 Mar 2019 01:47:00 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790124#comment-16790124
] 

yeshuangshuang commented on ZOOKEEPER-3211:
-------------------------------------------

[~hanm][~maoling]
The latest development, it is found that long living connections of zk will frequently print
the following log when something goes wrong:

2019-03-05 20:11:13,214 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:43911; will be dropped if server is in r-o
mode
 2019-03-05 20:11:18,817 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349]
- caught end of stream exception
 EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0058,
likely client has closed socket
 at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
 at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
 at java.lang.Thread.run(Thread.java:748)
 2019-03-05 20:11:19,818 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44131; will be dropped if server is in r-o
mode
 2019-03-05 20:11:19,881 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349]
- caught end of stream exception
 EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0000,
likely client has closed socket
 at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
 at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
 at java.lang.Thread.run(Thread.java:748)
 2019-03-05 20:11:19,882 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44135; will be dropped if server is in r-o
mode
 2019-03-05 20:11:20,079 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349]
- caught end of stream exception
 EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0057,
likely client has closed socket
 at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
 at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
 at java.lang.Thread.run(Thread.java:748)


> zookeeper standalone mode,found a high level bug in kernel of centos7.0 ,zookeeper
Server's  tcp/ip socket connections(default 60 ) are CLOSE_WAIT ,this lead to zk can't work
for client any more
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: ZOOKEEPER-3211
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3211
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.4.5
>         Environment: 1.zoo.cfg
> server.1=127.0.0.1:2902:2903
> 2.kernel
> kernel:Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Tue Feb 12 19:44:50
EST 2019 x86_64 x86_64 x86_64 GNU/Linux
> JDK:
> java version "1.7.0_181"
> OpenJDK Runtime Environment (rhel-2.6.14.5.el7-x86_64 u181-b00)
> OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
> zk: 3.4.5
>            Reporter: yeshuangshuang
>            Priority: Blocker
>             Fix For: 3.4.5
>
>         Attachments: 1.log, 2018-12-09_124131.png, 2018-12-09_124210.png, 2018-12-09_132854.png,
2018-12-09_133017.png, 2018-12-09_133049.png, 2018-12-09_133111.png, 2018-12-09_133131.png,
2018-12-09_133150.png, 2018-12-09_133210.png, 2018-12-09_133229.png, 2018-12-09_133248.png,
2018-12-09_133320.png
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> 1.config--zoo.cfg
> server.1=127.0.0.1:2902:2903
> 2.kernel version
> version:Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Tue Feb 12 19:44:50
EST 2019 x86_64 x86_64 x86_64 GNU/Linux
> JDK:
> java version "1.7.0_181"
> OpenJDK Runtime Environment (rhel-2.6.14.5.el7-x86_64 u181-b00)
> OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
> zk: 3.4.5
> 3.bug details:
> Occasionally,But the recurrence probability is extremely high. At first, the read-write
timeout takes about 6s, and after a few minutes, all connections (including long ones) will
be CLOSE_WAIT state.
> 4.:Circumvention scheme: it is found that all connections become close_wait to restart
the zookeeper server side actively



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message