zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-3242) Add server side connecting throttling
Date Wed, 23 Jan 2019 17:39:00 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750267#comment-16750267

Hudson commented on ZOOKEEPER-3242:

FAILURE: Integrated in Jenkins build ZooKeeper-trunk #363 (See [https://builds.apache.org/job/ZooKeeper-trunk/363/])
ZOOKEEPER-3242: Add server side connecting throttling (andor: rev 668828503389f3e542b846019bace8b9842fa428)
* (add) zookeeper-server/src/main/java/org/apache/zookeeper/server/ClientCnxnLimitException.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/admin/Commands.java
* (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
* (add) zookeeper-server/src/main/java/org/apache/zookeeper/server/BlueThrottle.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxn.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxnFactory.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServerMXBean.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/NIOServerCnxn.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerMetrics.java
* (edit) zookeeper-server/src/test/java/org/apache/zookeeper/server/admin/CommandsTest.java
* (add) zookeeper-server/src/test/java/org/apache/zookeeper/server/BlueThrottleTest.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServerBean.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/NIOServerCnxnFactory.java

> Add server side connecting throttling
> -------------------------------------
>                 Key: ZOOKEEPER-3242
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3242
>             Project: ZooKeeper
>          Issue Type: Improvement
>          Components: server
>            Reporter: Jie Huang
>            Assignee: Jie Huang
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 3.6.0
>          Time Spent: 3h
>  Remaining Estimate: 0h
> On-going performance investigation at Facebook has demonstrated that Zookeeper is easily
overwhelmed by spikes in connection rates and/or write request rates. Zookeeper performance
gets progressively worse, clients timeout and try to reconnect (exacerbating the problem)
and things enter a death spiral. To solve this problem, we need to add load protection to
Zookeeper via rate limiting and work shedding.
> This Jira adds a new connection rate limiting mechanism to Zookeeper in hopes of preventing
Zookeeper from becoming overwhelmed during connection spikes. 
> The new throttle is focused on limiting connections per second. The throttle is implemented
as a token-bucket with optional probabilistic dropping based on the BLUE queue management
> This token-bucket design allows the throttle to allow short bursts to pass, while still
capping the total number of requests per second. However, an issue with a token bucket approach
is that the wall clock arrival time of requests affects the probability of a request being
allowed to pass or not. Under constant load this can lead to request starvation for requests
that constantly arrive later than the majority. The optional probabilistic dropping mechanism
is designed to combat this, making rejections a random event with little skew based on arrival
> A more verbose description can be found in the comments in org.apache.zookeeper.server.BlueThrottle.
> By default, both the token-bucket and probabilistic dropping mechanism are disabled.
Enabling and tuning the throttles can be done both via Java system properties as well as against
a running node via JMX.
> The throttle has been tested and benchmarked at Facebook.

This message was sent by Atlassian JIRA

View raw message