zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-2701) Timeout for RecvWorker is too long
Date Mon, 16 Jul 2018 07:49:00 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544911#comment-16544911

Tom Lee commented on ZOOKEEPER-2701:

I can't speak to whether that solution is correct (suspect it's likely more complicated than
simply removing the code that modifies the read timeout for all sorts of reasons), but I can
imagine how the read timeout firing may address some of the symptoms by forcing the socket
closed & short-circuiting the OS-level retransmission guff.

Whatever the outcome, thanks for posting this bug! :) It was very helpful for tracking down
what was going on & helped us nail down our own system-level work-around (the tcp_retries2
thing seems to work great)

> Timeout for RecvWorker is too long
> ----------------------------------
>                 Key: ZOOKEEPER-2701
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2701
>             Project: ZooKeeper
>          Issue Type: Bug
>    Affects Versions: 3.4.8, 3.4.9, 3.4.10, 3.4.11
>         Environment: Centos6.5
> ZooKeeper 3.4.8
>            Reporter: Jiafu Jiang
>            Priority: Major
> Environment:
>  I deploy ZooKeeper in a cluster of three nodes. Each node has three network interfaces(eth0,
eth1, eth2).
> Hostname is used instead of IP address in zoo.cfg, and quorumListenOnAllIPs=true
> Probleam:
>  I start three ZooKeeper servers( node A, node B, and node C) one by one, 
>  when the leader election finishes, node B is the leader. 
>  Then I shutdown one network interface of node A by command "ifdown eth0". The ZooKeeper
server on node A will lost connection to node B and node C. In my test, I will take about
20 minites that the ZooKeepr server of node A realizes the event and try to call the QuorumServer.recreateSocketAddress
the resolve the hostname.
> I try to read the source code, and I find the code in
> {code:java|title=QuorumCnxManager.java:|borderStyle=solid}
>     class RecvWorker extends ZooKeeperThread {
>         Long sid;
>         Socket sock;
>         volatile boolean running = true;
>         final DataInputStream din;
>         final SendWorker sw;
>         RecvWorker(Socket sock, DataInputStream din, Long sid, SendWorker sw) {
>             super("RecvWorker:" + sid);
>             this.sid = sid;
>             this.sock = sock;
>             this.sw = sw;
>             this.din = din;
>             try {
>                 // OK to wait until socket disconnects while reading.
>                 sock.setSoTimeout(0);
>             } catch (IOException e) {
>                 LOG.error("Error while accessing socket for " + sid, e);
>                 closeSocket(sock);
>                 running = false;
>             }
>         }
>        ...
>      }
> {code}
> I notice that the soTime is set to 0 in RecvWorker constructor. I think this is reasonable
when the IP address of a ZooKeeper server never change, but considering that the IP address
of each ZooKeeper server may change, maybe we should better set a timeout here.
> I think this is a problem.

This message was sent by Atlassian JIRA

View raw message