Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Subject: Re: Zk OOM in Critical Thread
MIME-Version: 1.0
From: Chris Nauroth <cnauroth@hortonworks.com>
To: "user@zookeeper.apache.org" <user@zookeeper.apache.org>
CC: "Gupta, Abhishek" <ABHI.GUPTA@MorganStanley.com>, "Hejj, Botond"
	<Botond.Hejj@MorganStanley.com>
Thread-Topic: Zk OOM in Critical Thread
Thread-Index: AdCSRfDLfVVs8fJRRRCpbUcVPZXd3QADFIIA
Date: Tue, 19 May 2015 16:38:42 +0000
Message-ID: <D180B2E7.2295D%cnauroth@hortonworks.com>
References: <BD7421760F9E1F45975896E73667C1C70EABDF2C@HZWEX2010N4.msad.ms.com>
In-Reply-To: 
 <BD7421760F9E1F45975896E73667C1C70EABDF2C@HZWEX2010N4.msad.ms.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <AFBD5F794A63514A83FABED42E77B595@exch080.serverpod.net>
Content-Transfer-Encoding: quoted-printable

Hello Austin,

Yes, the ZooKeeper dev community has reacted to this by porting an
existing fix from trunk to the 3.4 code line.  The fix has been targeted
to the upcoming 3.4.7 release.  For more details, see ZOOKEEPER-602.

https://issues.apache.org/jira/browse/ZOOKEEPER-602


Additionally, we're going to recommend a best practice of killing the
server on OutOfMemoryError.  It sounds like you've already done this.
ZOOKEEPER-2185 tracks the necessary updates to documentation and scripts.

https://issues.apache.org/jira/browse/ZOOKEEPER-2185


The documentation already advised running the server under a process
supervisor, so terminating on OutOfMemoryError will cause the process
supervisor to initiate a restart.  Leadership will transition to another
node in the cluster.

I haven't personally seen an instance of ZooKeeper throwing
OutOfMemoryError for compressed class space, so I don't have any specific
advice on that.  Maybe others could respond if they've seen that.

--Chris Nauroth


On 5/19/15, 8:10 AM, "Miller, Austin" <Austin.Miller@morganstanley.com>
wrote:

>Hi all,
>
>We had an event in our prod cluster where an OOM caused a leader node to
>effectively become corrupted while the rest of the ensemble thought it
>was healthy, thus permanently degrading the ensemble to provide read only
>service on existing sessions until a human intervented.
>
>Exceptions in Critical Threads
>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
>As a tactical step, we've added an OOMHandler to bounce the node.
>However, we're cognizant of the fact that other exceptions in this space
>can cause this issue again.  There is also an interesting interaction
>with J8 which I will get to shortly.
>
>In this link:=20
>http://arstechnica.com/information-technology/2015/05/the-discovery-of-apa
>che-zookeepers-poison-packet/  (specifically bug #1) seems to apply to
>this issue.  I haven't extensively gone through the server code in some
>time, but will again shortly.  I'm wondering if this is seen as an issue
>by the zookeeper dev community and if there are plans to respond.
>
>OS: linux 64 bit
>Zk: 3.4.6
>jre: 1.8.31
>
>2015-05-10 19:11:49,882 - ERROR
>[QuorumPeer[myid=3D1]/0:0:0:0:0:0:0:0:2281:NIOServerCnxnFactory$1@44] -
>Thread Thread[QuorumPeer[myid=3D1]/0:0:0:0:0:0:0:0:2281,5,main] died
>
>java.lang.OutOfMemoryError: Compressed class space
>        at java.lang.ClassLoader.defineClass1(Native Method)
>        at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>        at=20
>java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>        at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
>        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>        at=20
>org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:6
>05)
>        at=20
>org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:798)
>
>Zookeeper and J8
>So while this all was occurring, the CCS space in J8 filled up.  This
>space is, by default, 1G.  For it to fill up feels surprising.  Maybe it
>was somehow due to lots of connections occurring.  This caused the OOM
>which caused the error in the leader thread.  I can't imagine what ZK
>server is doing to legitimately fill this space without instrumentation
>being involved somehow.  Or maybe J8 has a bug.  Any ideas on this would
>be appreciated.
>Austin
>
>
>________________________________
>
>NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>opinions or views contained herein are not intended to be, and do not
>constitute, advice within the meaning of Section 975 of the Dodd-Frank
>Wall Street Reform and Consumer Protection Act. If you have received this
>communication in error, please destroy all electronic and paper copies;
>do not disclose, use or act upon the information; and notify the sender
>immediately. Mistransmission is not intended to waive confidentiality or
>privilege. Morgan Stanley reserves the right, to the extent permitted
>under applicable law, to monitor electronic communications. This message
>is subject to terms available at the following link:
>http://www.morganstanley.com/disclaimers If you cannot access these
>links, please notify us by reply message and we will send the contents to
>you. By messaging with Morgan Stanley you consent to the foregoing.