zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <fpjunque...@yahoo.com.INVALID>
Subject Re: Consistently running out of heap space
Date Fri, 05 Sep 2014 13:47:25 GMT
Brian,

How much state are you storing in ZK? Can you check the size of the snapshots?

One common problem when folks are testing is that they forget to delete the data from previous
tests, so the state keeps accumulating and the server keeps crashing because the state is
too large.

Also, consider trying 3.4.5 just to see if it is a problem with 3.4.6 alone.

-Flavio


On Friday, September 5, 2014 2:23 PM, Brian C. Huffman <bhuffman@etinternational.com>
wrote:
 

>
>
>We're running the latest version of the stable 3.4 branch (3.4.6) and 
>have been consistently having problems running out of heap space.
>
>We're running a single server (redundancy isn't a concern at this point) 
>and I've tried the defaults (which seems to use Java's default heap of 
>8GB) as well as limiting to 3GB.  Either way the Zookeeper server 
>eventually dies.  With larger heap size it seems to take longer to die.
>
>Here's the latest trace:
>2014-09-05 00:51:11,419 [myid:] - ERROR 
>[SyncThread:0:SyncRequestProcessor@183] - Severe unrecoverable error, 
>exiting
>java.lang.OutOfMemoryError: Java heap space
>     at java.util.Arrays.copyOf(Arrays.java:2271)
>     at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
>     at 
>java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
>     at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
>     at java.io.DataOutputStream.write(DataOutputStream.java:107)
>     at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
>     at 
>org.apache.jute.BinaryOutputArchive.writeBuffer(BinaryOutputArchive.java:119)
>     at org.apache.zookeeper.txn.Txn.serialize(Txn.java:49)
>     at 
>org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
>     at org.apache.zookeeper.txn.MultiTxn.serialize(MultiTxn.java:44)
>     at 
>org.apache.zookeeper.server.persistence.Util.marshallTxnEntry(Util.java:263)
>     at 
>org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:216)
>     at 
>org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:314)
>     at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:476)
>     at 
>org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:140)
>2014-09-05 00:51:07,866 [myid:] - WARN 
>[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught 
>end of stream exception
>EndOfStreamException: Unable to read additional data from client 
>sessionid 0x14837ac98960071, likely client has closed socket
>     at 
>org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>     at 
>org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>     at java.lang.Thread.run(Thread.java:745)
>
>Here's my configuration:
>[user@xyz conf]$ grep -v '^#' zoo.cfg
>tickTime=2000
>initLimit=10
>syncLimit=5
>dataDir=/usr/local/var/zookeeper
>clientPort=2181
>autopurge.snapRetainCount=3
>autopurge.purgeInterval=1
>
>Can anyone suggest what the issue could be?
>
>Thanks,
>Brian
>
>
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message