Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
From: Mike Solomon <msolo@dropbox.com>
Date: Wed, 12 Oct 2016 12:57:54 -0700
Message-ID: <CA+dNya0eL-HVLjDpoQ8k-mtVc6Mg1-1y0AStV_tYbE=qOQD4yA@mail.gmail.com>
Subject: outstandingChanges queue grows without bound
To: user@zookeeper.apache.org
Content-Type: text/plain; charset=UTF-8
archived-at: Wed, 12 Oct 2016 20:06:37 -0000

I've been performance testing 3.5.2 and hit an interesting unavailability issue.

When there server is very busy (64k connections, 16k writes per
second) the leader can get busy enough that connections get throttled.
Enough throttling causes sessions to expire. As sessions expire, the
CPU consumption rises and the quorum is effectively unavailable.
Interestingly, if you shut down all the clients, the quorum won't heal
for nearly 10 minutes.

The issue is that the outstandingChanges queue has 250k items in it
and the closeSession code scans this linearly under a lock. Replacing
the linear scan with a hash table lookup improves this, but likely the
real solution is some backpressure on clients as a result of an
oversized outstandingChanges queue.

Here is a sample fix:
https://github.com/msolo/zookeeper/commit/75da352d506c2e3b0001d28acc058c422b3c8f0c

This results in the quorum healing about 30 seconds after the clients
disconnect.

Is there a way to prevent runaway growth in this queue? I'm wondering
if changing the definition of "throttling" to take into account the
size of this queue might help mitigate this. The end goal is that some
stable amount of traffic is reached asymptotically without suffering a
collapse.

Thanks,
-Mike