zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miller, Austin" <Austin.Mil...@morganstanley.com>
Subject RE: Zk OOM in Critical Thread
Date Tue, 19 May 2015 16:54:43 GMT
Hi Chris,

Thank you for the excellent response.

Regarding the J8 CCS issue, we're interested in why this happened.  Do you know if ZK server
off the top of your head is doing any sort of instrumentation?

In our situation, maybe thousands of attempted connections occurred on a node during an ensemble
election that maybe went on for a particularly long time.  Possibly network congestion prevented
the ensemble from reaching agreement in a timely fashion.  I have a small hypothesis that
ZK Server doesn't clean up timed out connections during that time and that this somehow fills
CCS space.  I have taken steps to try to reproduce this exact situation but it proves to be
difficult.

Austin

-----Original Message-----
From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
Sent: Tuesday, May 19, 2015 12:39 PM
To: user@zookeeper.apache.org
Cc: Gupta, Abhishek (ICT); Hejj, Botond (Enterprise Infrastructure)
Subject: Re: Zk OOM in Critical Thread

Hello Austin,

Yes, the ZooKeeper dev community has reacted to this by porting an
existing fix from trunk to the 3.4 code line.  The fix has been targeted
to the upcoming 3.4.7 release.  For more details, see ZOOKEEPER-602.

https://issues.apache.org/jira/browse/ZOOKEEPER-602


Additionally, we're going to recommend a best practice of killing the
server on OutOfMemoryError.  It sounds like you've already done this.
ZOOKEEPER-2185 tracks the necessary updates to documentation and scripts.

https://issues.apache.org/jira/browse/ZOOKEEPER-2185


The documentation already advised running the server under a process
supervisor, so terminating on OutOfMemoryError will cause the process
supervisor to initiate a restart.  Leadership will transition to another
node in the cluster.

I haven't personally seen an instance of ZooKeeper throwing
OutOfMemoryError for compressed class space, so I don't have any specific
advice on that.  Maybe others could respond if they've seen that.

--Chris Nauroth




On 5/19/15, 8:10 AM, "Miller, Austin" <Austin.Miller@morganstanley.com>
wrote:

>Hi all,
>
>We had an event in our prod cluster where an OOM caused a leader node to
>effectively become corrupted while the rest of the ensemble thought it
>was healthy, thus permanently degrading the ensemble to provide read only
>service on existing sessions until a human intervented.
>
>Exceptions in Critical Threads
>============
>
>As a tactical step, we've added an OOMHandler to bounce the node.
>However, we're cognizant of the fact that other exceptions in this space
>can cause this issue again.  There is also an interesting interaction
>with J8 which I will get to shortly.
>
>In this link:
>http://arstechnica.com/information-technology/2015/05/the-discovery-of-apa
>che-zookeepers-poison-packet/  (specifically bug #1) seems to apply to
>this issue.  I haven't extensively gone through the server code in some
>time, but will again shortly.  I'm wondering if this is seen as an issue
>by the zookeeper dev community and if there are plans to respond.
>
>OS: linux 64 bit
>Zk: 3.4.6
>jre: 1.8.31
>
>2015-05-10 19:11:49,882 - ERROR
>[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2281:NIOServerCnxnFactory$1@44] -
>Thread Thread[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2281,5,main] died
>
>java.lang.OutOfMemoryError: Compressed class space
>        at java.lang.ClassLoader.defineClass1(Native Method)
>        at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>        at
>java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>        at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
>        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>        at
>org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:6
>05)
>        at
>org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:798)
>
>Zookeeper and J8
>So while this all was occurring, the CCS space in J8 filled up.  This
>space is, by default, 1G.  For it to fill up feels surprising.  Maybe it
>was somehow due to lots of connections occurring.  This caused the OOM
>which caused the error in the leader thread.  I can't imagine what ZK
>server is doing to legitimately fill this space without instrumentation
>being involved somehow.  Or maybe J8 has a bug.  Any ideas on this would
>be appreciated.
>Austin
>
>
>________________________________
>
>NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>opinions or views contained herein are not intended to be, and do not
>constitute, advice within the meaning of Section 975 of the Dodd-Frank
>Wall Street Reform and Consumer Protection Act. If you have received this
>communication in error, please destroy all electronic and paper copies;
>do not disclose, use or act upon the information; and notify the sender
>immediately. Mistransmission is not intended to waive confidentiality or
>privilege. Morgan Stanley reserves the right, to the extent permitted
>under applicable law, to monitor electronic communications. This message
>is subject to terms available at the following link:
>http://www.morganstanley.com/disclaimers If you cannot access these
>links, please notify us by reply message and we will send the contents to
>you. By messaging with Morgan Stanley you consent to the foregoing.



--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained
herein are not intended to be, and do not constitute, advice within the meaning of Section
975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received
this communication in error, please destroy all electronic and paper copies; do not disclose,
use or act upon the information; and notify the sender immediately. Mistransmission is not
intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the
extent permitted under applicable law, to monitor electronic communications. This message
is subject to terms available at the following link: http://www.morganstanley.com/disclaimers.
If you cannot access these links, please notify us by reply message and we will send the contents
to you. By messaging with Morgan Stanley you consent to the foregoing.

Mime
View raw message