Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (nike.apache.org: domain of brian@nimbula.com designates
 209.85.212.42 as permitted sender)
MIME-Version: 1.0
Date: Wed, 23 May 2012 17:41:08 -0700
Message-ID: 
 <CA+iORZVZq7AyQoAjfPUhYoSivwPodnyYqdwGJMEOa1nN5ih4iA@mail.gmail.com>
Subject: ZooKeeper's resident set size grows but does not shrink
From: Brian Oki <brian@nimbula.com>
To: user@zookeeper.apache.org
Cc: Brian Oki <brian@nimbula.com>
Content-Type: multipart/alternative; boundary=f46d043c81ccf11a5d04c0bd8152

--f46d043c81ccf11a5d04c0bd8152
Content-Type: text/plain; charset=ISO-8859-1

Hello,

We use ZooKeeper 3.3.3.  On a 3-node site, we've been using Patrick Hunt's
publicly available latencies test suite to create scenarios that will help
us to understand the memory, CPU and disk requirements for a deployment of
ZooKeeper for our type of workload.  We use a fourth node as the ZooKeeper
(ZK) client to conduct the tests.

We modified zk-latencies.py slightly to just create-set-delete znodes only.
 In particular, we create 1000 permanent znodes, each written with 250,000
bytes of data.  We do this create-set-delete in a loop, sleeping for 5
seconds between iterations.

We observe at the ZK leader that the Resident Set Size (RSS) memory climbs
rapidly to 2.6 GB on an 8 GB RAM node.  The Java heap size of each ZK
server daemon is 3 GB.

Further, once the test has gone through 15 iterations, all the znodes
created on behalf of the test have been deleted.  There is no further write
activity to ZK, and no read activity at all.  The system is quiesced.  No
other services are competing for the disk, CPU or RAM during the test.

Our question is this: The RSS of the ZK leader (and the followers) seems to
remain at 2.6 GB after the test has completed.  Why?

We would expect that since all relevant znodes for the test have been
deleted, the leader's RSS should have shrunk considerably, even after 1
hour has passed.  Are we missing something?

We have used jmap to inspect the heap.  To understand the heap contents
requires detailed implementation knowledge that we don't have, so we didn't
pursue this avenue any further.

Configuration:
   3 node servers running ZK daemons as 3-server ensemble
   1 client machine
   each node has 8 GB RAM
   each node has 4 cores
   each node has a 465 GB disk
   ZK release: 3.3.3
   ZK server java heap size: 3 GB
  GC: concurrent low-pause garbage collector
   NIC: bonded 1 Gb NIC

Thank you.

Sincerely,

Brian

--f46d043c81ccf11a5d04c0bd8152--