I notice that: there are more than 100 "CLOSE_WAIT" incomming connections on storage port 7000 In my two cassandra node: 126 of 146 storage connections is "CLOSE_WAIT" 196 of 217 storage connections is "CLOSE_WAIT" Is it normal? -------------------------------------------------- From: "Chris Goffinet" Sent: Friday, June 04, 2010 1:50 PM To: Cc: Subject: Re: High CPU Usage since 0.6.2 > We're seeing this as well. We were testing with a 40+ node cluster on the > latest 0.6 branch from few days ago. > > -Chris > > On Jun 3, 2010, at 9:55 PM, Lu Ming wrote: > >> >> I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to >> cassandra to 0.6.2 yesterday. >> But today I find six cassandra nodes have high CPU usage more than 400% >> in my 8-core CPU sever. >> The worst one is more than 760%. It is very serious. >> >> I use jvisualvm to watch the worst node, and I found that there are many >> running threads named "thread-xxx" >> the status of other threads is waiting and sleeping. >> >> "Thread-130" - Thread t@240 >> java.lang.Thread.State: RUNNABLE >> at sun.misc.Unsafe.setMemory(Native Method) >> at sun.nio.ch.Util.erase(Util.java:202) >> at >> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560) >> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) >> at >> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) >> Locked ownable synchronizers: >> - None >> >> "Thread-126" - Thread t@236 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.FileDispatcher.read0(Native Method) >> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) >> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) >> at sun.nio.ch.IOUtil.read(IOUtil.java:200) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) >> - locked java.lang.Object@10808561 >> at >> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565) >> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) >> at >> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) >> >> Locked ownable synchronizers: >> - None >> >> "Thread-119" - Thread t@229 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.NativeThread.current(Native Method) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182) >> - locked java.lang.Object@65b4abbd >> - locked java.lang.Object@38773975 >> at >> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565) >> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) >> at >> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) >> >> Locked ownable synchronizers: >> - None >> >> > >