Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (nike.apache.org: domain of fpjunqueira@yahoo.com
 designates 98.139.213.74 as permitted sender)
References: <5409B95D.7070807@etinternational.com>
Message-ID: <1409924845.82416.YahooMailNeo@web142304.mail.bf1.yahoo.com>
Date: Fri, 5 Sep 2014 06:47:25 -0700
From: Flavio Junqueira <fpjunqueira@yahoo.com.INVALID>
Reply-To: Flavio Junqueira <fpjunqueira@yahoo.com>
Subject: Re: Consistently running out of heap space
To: "user@zookeeper.apache.org" <user@zookeeper.apache.org>
In-Reply-To: <5409B95D.7070807@etinternational.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="2138815387-1226932479-1409924845=:82416"

--2138815387-1226932479-1409924845=:82416
Content-Type: text/plain; charset=us-ascii

Brian,

How much state are you storing in ZK? Can you check the size of the snapshots?

One common problem when folks are testing is that they forget to delete the data from previous tests, so the state keeps accumulating and the server keeps crashing because the state is too large.

Also, consider trying 3.4.5 just to see if it is a problem with 3.4.6 alone.

-Flavio


On Friday, September 5, 2014 2:23 PM, Brian C. Huffman <bhuffman@etinternational.com> wrote:
 

>
>
>We're running the latest version of the stable 3.4 branch (3.4.6) and 
>have been consistently having problems running out of heap space.
>
>We're running a single server (redundancy isn't a concern at this point) 
>and I've tried the defaults (which seems to use Java's default heap of 
>8GB) as well as limiting to 3GB.  Either way the Zookeeper server 
>eventually dies.  With larger heap size it seems to take longer to die.
>
>Here's the latest trace:
>2014-09-05 00:51:11,419 [myid:] - ERROR 
>[SyncThread:0:SyncRequestProcessor@183] - Severe unrecoverable error, 
>exiting
>java.lang.OutOfMemoryError: Java heap space
>     at java.util.Arrays.copyOf(Arrays.java:2271)
>     at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
>     at 
>java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
>     at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
>     at java.io.DataOutputStream.write(DataOutputStream.java:107)
>     at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
>     at 
>org.apache.jute.BinaryOutputArchive.writeBuffer(BinaryOutputArchive.java:119)
>     at org.apache.zookeeper.txn.Txn.serialize(Txn.java:49)
>     at 
>org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
>     at org.apache.zookeeper.txn.MultiTxn.serialize(MultiTxn.java:44)
>     at 
>org.apache.zookeeper.server.persistence.Util.marshallTxnEntry(Util.java:263)
>     at 
>org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:216)
>     at 
>org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:314)
>     at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:476)
>     at 
>org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:140)
>2014-09-05 00:51:07,866 [myid:] - WARN 
>[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught 
>end of stream exception
>EndOfStreamException: Unable to read additional data from client 
>sessionid 0x14837ac98960071, likely client has closed socket
>     at 
>org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>     at 
>org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>     at java.lang.Thread.run(Thread.java:745)
>
>Here's my configuration:
>[user@xyz conf]$ grep -v '^#' zoo.cfg
>tickTime=2000
>initLimit=10
>syncLimit=5
>dataDir=/usr/local/var/zookeeper
>clientPort=2181
>autopurge.snapRetainCount=3
>autopurge.purgeInterval=1
>
>Can anyone suggest what the issue could be?
>
>Thanks,
>Brian
>
>
>
>
--2138815387-1226932479-1409924845=:82416--