zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Han (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ZOOKEEPER-3399) Remove logging in getGlobalOutstandingLimit for optimal performance.
Date Wed, 22 May 2019 01:43:00 GMT
Michael Han created ZOOKEEPER-3399:

             Summary: Remove logging in getGlobalOutstandingLimit for optimal performance.
                 Key: ZOOKEEPER-3399
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3399
             Project: ZooKeeper
          Issue Type: Bug
          Components: server
    Affects Versions: 3.6.0
            Reporter: Michael Han
            Assignee: Michael Han

Recently we have moved some of our production clusters to the top of the trunk. One issue
we found is a performance regression on read and write latency on the clusters where the quorum
is also serving traffic. The average read latency increased by 50x, p99 read latency increased
by 300x. 

The root cause is a log statement introduced in ZOOKEEPER-3177 (PR711), where we added a LOG.info
statement in getGlobalOutstandingLimit. getGlobalOutstandingLimit is on the critical code
path for request processing and for each request, it will be called twice (one at processing
the packet, one at finalizing the request response). This not only degrades performance of
the server, but also bloated the log file, when the QPS of a server is high.

This only impacts clusters when the quorum (leader + follower) is serving traffic. For clusters
where only observers are serving traffic no impact is observed.



This message was sent by Atlassian JIRA

View raw message