zookeeper-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-3356) Request throttling in Netty is not working as expected and could cause direct buffer OOM issue
Date Wed, 24 Jul 2019 04:39:00 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891602#comment-16891602

Hudson commented on ZOOKEEPER-3356:

SUCCESS: Integrated in Jenkins build Zookeeper-trunk-single-thread #461 (See [https://builds.apache.org/job/Zookeeper-trunk-single-thread/461/])
ZOOKEEPER-3356: Implement advanced Netty flow control based on feedback (eolivelli: rev ce33b7faed60ea0b7c6f2eb1edbf56eec20a8bc2)
* (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
* (edit) zookeeper-server/src/test/java/org/apache/zookeeper/server/NettyServerCnxnTest.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerMetrics.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxnFactory.java
* (edit) zookeeper-server/src/test/java/org/apache/zookeeper/test/SSLAuthTest.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxn.java

> Request throttling in Netty is not working as expected and could cause direct buffer
OOM issue 
> -----------------------------------------------------------------------------------------------
>                 Key: ZOOKEEPER-3356
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3356
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.5.4, 3.6.0
>            Reporter: Fangmin Lv
>            Assignee: Fangmin Lv
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.6.0
>          Time Spent: 3h
>  Remaining Estimate: 0h
> The current implementation of Netty enable/disable recv logic may cause the direct buffer
OOM because we may enable read a large chunk of packets and disabled again after consuming
a single ZK request. We have seen this problem on prod occasionally.
> Need a more advanced flow control in Netty instead of using AUTO_READ. Have improved it
internally by enable/disable recv based on the queuedBuffer size, will upstream this soon.
> With this implementation, the max Netty queued buffer size (direct memory usage) will
be 2 * recv_buffer size. It's not the per message size because in epoll ET mode it will try
to read until the socket is empty, and because of SslHandler will trigger another read when
it's not a full encrypt packet and haven't issued any decrypt message.

This message was sent by Atlassian JIRA

View raw message