Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates
 209.85.215.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=tA0fNvdKSNjOsplwtYOCoS9clDVYzCWwoQ5m6T6ymb5qjADIJ43HkfL8x/2S9d9HOD
         xq0FNVqWrhO2K1GjcedNhINg8hAl2B9dFkCxDqx1Cl89FhhB+9TNitJWLWPoqs4xMiJs
         xadegIRk9O1oX+IVvFUmZPxc6sNTG5w0cjVXo=
MIME-Version: 1.0
In-Reply-To: <AANLkTinbuaCsR6Xyt4s11Lv4wn17T-ef-0_KAbVaAqww@mail.gmail.com>
References: <AANLkTinbuaCsR6Xyt4s11Lv4wn17T-ef-0_KAbVaAqww@mail.gmail.com>
Date: Wed, 20 Oct 2010 16:16:53 -0500
Message-ID: <AANLkTimzndUJOimNwJWpohL0B68bF6xSEEMvdVvW1YP=@mail.gmail.com>
Subject: Re: Cassandra crashed - possible JMX threads leak
From: Jonathan Ellis <jbellis@gmail.com>
To: user <user@cassandra.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

can you reproduce this by, say, running nodeprobe ring in a bash while loop=
?

On Wed, Oct 20, 2010 at 3:09 PM, Bill Au <bill.w.au@gmail.com> wrote:
> One of my Cassandra server crashed with the following:
>
> ERROR [ACCEPT-xxx.xxx.xxx/nnn.nnn.nnn.nnn] 2010-10-19 00:25:10,419
> CassandraDaemon.java (line 82) Uncaught exception in thread
> Thread[ACCEPT-xxx.xxx.xxx/nnn.nnn.nnn.nnn,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.start0(Native Method)
> =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.start(Thread.java:597)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingServi=
ce.java:533)
>
>
> I took threads dump in the JVM on all the other Cassandra severs in my
> cluster.=A0 They all have thousand of threads looking like this:
>
> "JMX server connection timeout 183373" daemon prio=3D10 tid=3D0x00002aad2=
30db800
> nid=3D0x5cf6 in Object.wait() [0x00002aad7a316000]
> =A0=A0 java.lang.Thread.State: TIMED_WAITING (on object monitor)
> =A0=A0=A0=A0=A0=A0=A0 at java.lang.Object.wait(Native Method)
> =A0=A0=A0=A0=A0=A0=A0 at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCom=
municatorAdmin.java:150)
> =A0=A0=A0=A0=A0=A0=A0 - locked <0x00002aab056ccee0> (a [I)
> =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:619)
>
> It seems to me that there is a JMX threads leak in Cassandra.=A0 NodeProb=
e
> creates a JMXConnector but never calls its close() method.=A0 I tried set=
ting
> jmx.remote.x.server.connection.timeout to 0 hoping that would disable the
> JMX server connection timeout threads.=A0 But that did not make any
> difference.
>
> Has anyone else seen this?
>
> Bill
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com