zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Victor <victorfu...@yahoo.com>
Subject Memory leak in zookeeper 3.3.2 and 3.3.3?
Date Wed, 01 Jun 2011 06:00:36 GMT
Hi, 
I apologize for the broadcasting.  I searched the archive before I send this email to the
mailing list. 
 
 We are using Cages library + Zookeeper 3.3.3 to synchronize creation of forum user name
(which needs to be unique).  
  This is the only use case that we write in ZooKeeper. 

  We used Cages ZkWriteLock to obtain write lock and below is the code (which is very straightforward):
    org.wyki.zookeeper.cages.ZkWriteLock lock = new org.wyki.zookeeper.cages.ZkWriteLock("/User/ForumUsername/"
+ forumUsername);
            try {
                boolean lockAcquired = lock.acquire(5, TimeUnit.SECONDS);
    ......
            } finally {
                lock.release();
            }
   
  In our load test (1 single Zookeeper server), even with 3GB max heap size, ZooKeeper runs
out of memory after ~3 hours.
  So I decided to profile Zookeeper with YourKit Jave profile (9.5) against one Zookeeper
server.   After every 6 above calls to ZooKeeper (I randomly picked 6),  I saw the memory
usage increased ~200K (in Yourkit retained size increased ~220K and shallow size increased

  Even after 1 hour or more, even if I forced garbage collection (done in YourKit), the memory
increased due to the 6 calls didn't get released. I ran the 6 calls for a few times and observed
the same. So I suspect there is a memory leak (and that is why we got OutOfMemory in load
test)
  Looking further at the hotspot using Yourkit, the memory increase are in the form of String,
char[], Class, HashMap$Entry (java classes or types) and maily from below method invocation:
  
  org.apache.zookeeper.server.FinalRequestProcessor.processRequest(Request)
  org.apache.zookeeper.server.PrepRequestProcessor.pRequest(Request)
  org.apache.zookeeper.server.NIOServerCnxn$Factory.run()
  
  But from the code, leakage is not obvious.
  
  
  We used below JVM (for ZooKeeper) startup options/flags:
  -server -Xms1536m -Xmx3072m -Xloggc:/var/zookeeper/logs/gc.log -XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
-XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=54321 -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
  
    I looked at ZooKeeper documentation thoroughly (especially the administration guide),
but couldn't find a way to tune this (to avoid above suspected memory leak)
   
   Is there a memory leak in Zookeeper 3.3.3 (or 3.3.2)? If there is, How could we configure
ZooKeeper to avoid/reduce that leak?    What is the stable version to use? did we misconfigure
anything?
   
Please advise or help. Thanks a lot!
 
Victor
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message