zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yeshuangshuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-3211) zookeeper standalone mode,found a high level bug in kernel of centos7.0 ,zookeeper Server's tcp/ip socket connections(default 60 ) are CLOSE_WAIT ,this lead to zk can't work for client any more
Date Wed, 06 Mar 2019 02:40:00 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785170#comment-16785170
] 

yeshuangshuang commented on ZOOKEEPER-3211:
-------------------------------------------

        at java.lang.Thread.run(Thread.java:748)
2019-03-05 20:11:13,214 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:43911; will be dropped if server is in r-o
mode
2019-03-05 20:11:18,817 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349]
- caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0058,
likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:748)
2019-03-05 20:11:19,818 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44131; will be dropped if server is in r-o
mode
2019-03-05 20:11:19,881 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349]
- caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0000,
likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:748)
2019-03-05 20:11:19,882 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44135; will be dropped if server is in r-o
mode
2019-03-05 20:11:20,079 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349]
- caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0057,
likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:748)
2019-03-05 20:11:21,079 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44179; will be dropped if server is in r-o
mode
2019-03-05 20:11:24,517 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44284; will be dropped if server is in r-o
mode
2019-03-05 20:11:26,485 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349]
- caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0058,
likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:748)
2019-03-05 20:11:26,550 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44316; will be dropped if server is in r-o
mode
2019-03-05 20:11:27,485 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44358; will be dropped if server is in r-o
mode
2019-03-05 20:11:33,221 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44490; will be dropped if server is in r-o
mode
2019-03-05 20:11:33,746 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44512; will be dropped if server is in r-o
mode
2019-03-05 20:11:35,153 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44554; will be dropped if server is in r-o
mode
2019-03-05 20:11:39,888 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44670; will be dropped if server is in r-o
mode
2019-03-05 20:11:40,625 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44691; will be dropped if server is in r-o
mode
2019-03-05 20:11:42,821 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44763; will be dropped if server is in r-o
mode
2019-03-05 20:11:46,414 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44889; will be dropped if server is in r-o
mode
2019-03-05 20:11:46,556 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44893; will be dropped if server is in r-o
mode
2019-03-05 20:11:50,489 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:44952; will be dropped if server is in r-o
mode
2019-03-05 20:11:53,228 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45007; will be dropped if server is in r-o
mode
2019-03-05 20:11:58,156 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45151; will be dropped if server is in r-o
mode
2019-03-05 20:11:59,081 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45180; will be dropped if server is in r-o
mode
2019-03-05 20:11:59,897 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45206; will be dropped if server is in r-o
mode
2019-03-05 20:12:04,244 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45270; will be dropped if server is in r-o
mode
2019-03-05 20:12:05,824 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45300; will be dropped if server is in r-o
mode
2019-03-05 20:12:06,567 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45310; will be dropped if server is in r-o
mode
2019-03-05 20:12:10,373 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45422; will be dropped if server is in r-o
mode
2019-03-05 20:12:11,749 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45448; will be dropped if server is in r-o
mode
2019-03-05 20:12:13,237 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45491; will be dropped if server is in r-o
mode
2019-03-05 20:12:13,492 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45498; will be dropped if server is in r-o
mode
2019-03-05 20:12:19,905 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45582; will be dropped if server is in r-o
mode
2019-03-05 20:12:21,161 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45592; will be dropped if server is in r-o
mode
2019-03-05 20:12:24,367 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45657; will be dropped if server is in r-o
mode
2019-03-05 20:12:24,417 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45658; will be dropped if server is in r-o
mode
2019-03-05 20:12:26,574 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793]
- Connection request from old client /127.0.0.1:45700; will be dropped if server is 

zookeeper.log 

> zookeeper standalone mode,found a high level bug in kernel of centos7.0 ,zookeeper
Server's  tcp/ip socket connections(default 60 ) are CLOSE_WAIT ,this lead to zk can't work
for client any more
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: ZOOKEEPER-3211
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3211
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.4.5
>         Environment: 1.zoo.cfg
> server.1=127.0.0.1:2902:2903
> 2.kernel
> kernel:Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Tue Feb 12 19:44:50
EST 2019 x86_64 x86_64 x86_64 GNU/Linux
> JDK:
> java version "1.7.0_181"
> OpenJDK Runtime Environment (rhel-2.6.14.5.el7-x86_64 u181-b00)
> OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
> zk: 3.4.5
>            Reporter: yeshuangshuang
>            Priority: Blocker
>             Fix For: 3.4.5
>
>         Attachments: 1.log, 2018-12-09_124131.png, 2018-12-09_124210.png, 2018-12-09_132854.png,
2018-12-09_133017.png, 2018-12-09_133049.png, 2018-12-09_133111.png, 2018-12-09_133131.png,
2018-12-09_133150.png, 2018-12-09_133210.png, 2018-12-09_133229.png, 2018-12-09_133248.png,
2018-12-09_133320.png
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> 1.config--zoo.cfg
> server.1=127.0.0.1:2902:2903
> 2.kernel version
> version:Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Tue Feb 12 19:44:50
EST 2019 x86_64 x86_64 x86_64 GNU/Linux
> JDK:
> java version "1.7.0_181"
> OpenJDK Runtime Environment (rhel-2.6.14.5.el7-x86_64 u181-b00)
> OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
> zk: 3.4.5
> 3.bug details:
> Occasionally,But the recurrence probability is extremely high. At first, the read-write
timeout takes about 6s, and after a few minutes, all connections (including long ones) will
be CLOSE_WAIT state.
> 4.:Circumvention scheme: it is found that all connections become close_wait to restart
the zookeeper server side actively



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message