zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian C. Huffman" <bhuff...@etinternational.com>
Subject Re: Consistently running out of heap space
Date Fri, 05 Sep 2014 14:33:49 GMT
Flavio,

I was having the same problems on 3.4.5 so I upgraded to 3.4.6.  So it 
doesn't seem to be related to the version.

You might be right about the storing of state.  I'm curious - does the 
"state" consist of the entire node listing? Is there anyway to tell 
zookeeper to keep a node around but only on disk?

Thanks,
Brian

On 09/05/2014 09:47 AM, Flavio Junqueira wrote:
> Brian,
>
> How much state are you storing in ZK? Can you check the size of the snapshots?
>
> One common problem when folks are testing is that they forget to delete the data from
previous tests, so the state keeps accumulating and the server keeps crashing because the
state is too large.
>
> Also, consider trying 3.4.5 just to see if it is a problem with 3.4.6 alone.
>
> -Flavio
>
>
> On Friday, September 5, 2014 2:23 PM, Brian C. Huffman <bhuffman@etinternational.com>
wrote:
>   
>
>>
>> We're running the latest version of the stable 3.4 branch (3.4.6) and
>> have been consistently having problems running out of heap space.
>>
>> We're running a single server (redundancy isn't a concern at this point)
>> and I've tried the defaults (which seems to use Java's default heap of
>> 8GB) as well as limiting to 3GB.  Either way the Zookeeper server
>> eventually dies.  With larger heap size it seems to take longer to die.
>>
>> Here's the latest trace:
>> 2014-09-05 00:51:11,419 [myid:] - ERROR
>> [SyncThread:0:SyncRequestProcessor@183] - Severe unrecoverable error,
>> exiting
>> java.lang.OutOfMemoryError: Java heap space
>>      at java.util.Arrays.copyOf(Arrays.java:2271)
>>      at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
>>      at
>> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
>>      at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
>>      at java.io.DataOutputStream.write(DataOutputStream.java:107)
>>      at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
>>      at
>> org.apache.jute.BinaryOutputArchive.writeBuffer(BinaryOutputArchive.java:119)
>>      at org.apache.zookeeper.txn.Txn.serialize(Txn.java:49)
>>      at
>> org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
>>      at org.apache.zookeeper.txn.MultiTxn.serialize(MultiTxn.java:44)
>>      at
>> org.apache.zookeeper.server.persistence.Util.marshallTxnEntry(Util.java:263)
>>      at
>> org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:216)
>>      at
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:314)
>>      at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:476)
>>      at
>> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:140)
>> 2014-09-05 00:51:07,866 [myid:] - WARN
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught
>> end of stream exception
>> EndOfStreamException: Unable to read additional data from client
>> sessionid 0x14837ac98960071, likely client has closed socket
>>      at
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>>      at
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>>      at java.lang.Thread.run(Thread.java:745)
>>
>> Here's my configuration:
>> [user@xyz conf]$ grep -v '^#' zoo.cfg
>> tickTime=2000
>> initLimit=10
>> syncLimit=5
>> dataDir=/usr/local/var/zookeeper
>> clientPort=2181
>> autopurge.snapRetainCount=3
>> autopurge.purgeInterval=1
>>
>> Can anyone suggest what the issue could be?
>>
>> Thanks,
>> Brian
>>
>>
>>
>>



Mime
View raw message