cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lu Ming" <xl...@live.com>
Subject Re: High CPU Usage since 0.6.2
Date Fri, 04 Jun 2010 11:01:16 GMT
I do the Thread Dump on each cassandra node, and count the thread with call 
stack string "at 
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)atorg.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.jav

a:66)" in "thread-xxx"

then I find an interesting things.

Server Name/Thread Count/Average CPU Usage
A   0  <20%
B   8  760%
C   0  <20%
D   3  300%
E   4  220%
F   0  <20%
G   2  200%
H   7  700%
I   2  200%
J   3  300%

It seems that a thread calling 
"org.apache.cassandra.net.IncomingTcpConnection.run" occupies the 100% CPU
So I guess the code in IncomingTcpConnection.java:66 cause high CPU usage.

                if (isStream)
                {
                    new IncomingStreamReader(socket.getChannel()).read();
                }



--------------------------------------------------
From: "Lu Ming" <xluke@live.com>
Sent: Friday, June 04, 2010 12:55 PM
To: <cassandra-user@incubator.apache.org>
Subject: High CPU Usage since 0.6.2

>
> I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to 
> cassandra to 0.6.2 yesterday.
> But today I find six cassandra nodes have high CPU usage more than 400% in 
> my 8-core CPU sever.
> The worst one is more than 760%. It is very serious.
>
> I use jvisualvm to watch the worst node, and I found that there are many 
> running threads named "thread-xxx"
> the status of other threads is waiting and sleeping.
>
> "Thread-130" - Thread t@240
>   java.lang.Thread.State: RUNNABLE
> at sun.misc.Unsafe.setMemory(Native Method)
> at sun.nio.ch.Util.erase(Util.java:202)
> at 
> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>   Locked ownable synchronizers:
> - None
>
> "Thread-126" - Thread t@236
>   java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> - locked java.lang.Object@10808561
> at 
> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>
>   Locked ownable synchronizers:
> - None
>
> "Thread-119" - Thread t@229
>   java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.NativeThread.current(Native Method)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182)
> - locked java.lang.Object@65b4abbd
> - locked java.lang.Object@38773975
> at 
> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>
>   Locked ownable synchronizers:
> - None
>
>
>
> 

Mime
View raw message