incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lu Ming" <xl...@live.com>
Subject Re: High CPU Usage since 0.6.2
Date Fri, 04 Jun 2010 09:34:00 GMT
I notice that: there are more than 100 "CLOSE_WAIT"  incomming connections 
on storage port 7000

In my two cassandra node:
126 of 146 storage connections is "CLOSE_WAIT"
196 of 217 storage connections is "CLOSE_WAIT"

Is it normal?

--------------------------------------------------
From: "Chris Goffinet" <goffinet@digg.com>
Sent: Friday, June 04, 2010 1:50 PM
To: <user@cassandra.apache.org>
Cc: <cassandra-user@incubator.apache.org>
Subject: Re: High CPU Usage since 0.6.2

> We're seeing this as well. We were testing with a 40+ node cluster on the 
> latest 0.6 branch from few days ago.
>
> -Chris
>
> On Jun 3, 2010, at 9:55 PM, Lu Ming wrote:
>
>>
>> I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to 
>> cassandra to 0.6.2 yesterday.
>> But today I find six cassandra nodes have high CPU usage more than 400% 
>> in my 8-core CPU sever.
>> The worst one is more than 760%. It is very serious.
>>
>> I use jvisualvm to watch the worst node, and I found that there are many 
>> running threads named "thread-xxx"
>> the status of other threads is waiting and sleeping.
>>
>> "Thread-130" - Thread t@240
>>  java.lang.Thread.State: RUNNABLE
>> at sun.misc.Unsafe.setMemory(Native Method)
>> at sun.nio.ch.Util.erase(Util.java:202)
>> at 
>> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at 
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at 
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>>  Locked ownable synchronizers:
>> - None
>>
>> "Thread-126" - Thread t@236
>>  java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> - locked java.lang.Object@10808561
>> at 
>> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at 
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at 
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>>
>>  Locked ownable synchronizers:
>> - None
>>
>> "Thread-119" - Thread t@229
>>  java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.NativeThread.current(Native Method)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182)
>> - locked java.lang.Object@65b4abbd
>> - locked java.lang.Object@38773975
>> at 
>> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at 
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at 
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>>
>>  Locked ownable synchronizers:
>> - None
>>
>>
>
> 

Mime
View raw message