incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ingram Chen <ingramc...@gmail.com>
Subject busy thread on IncomingStreamReader ?
Date Thu, 15 Apr 2010 16:33:54 GMT
Hi all,

 We setup two nodes and simply set replication factor=2 for test run.

After both nodes, say, node A and node B, serve several hours, we found that
"node A" always keep 300% cpu usage.
(the other node is under 100% cpu, which is normal)

thread dump on "node A" shows that there are 3 busy threads related to
IncomingStreamReader:

==========================

"Thread-66" prio=10 tid=0x00002aade4018800 nid=0x69e7 runnable
[0x000000004030a000]
   java.lang.Thread.State: RUNNABLE
        at sun.misc.Unsafe.setMemory(Native Method)
        at sun.nio.ch.Util.erase(Util.java:202)
        at
sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
        at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
        at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
        at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)

"Thread-65" prio=10 tid=0x00002aade4017000 nid=0x69e6 runnable
[0x000000004d44b000]
   java.lang.Thread.State: RUNNABLE
        at sun.misc.Unsafe.setMemory(Native Method)
        at sun.nio.ch.Util.erase(Util.java:202)
        at
sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
        at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
        at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
        at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)

"Thread-62" prio=10 tid=0x00002aade4014800 nid=0x4150 runnable
[0x000000004d34a000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.FileChannelImpl.size0(Native Method)
        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:309)
        - locked <0x00002aaac450dcd0> (a java.lang.Object)
        at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:597)
        at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
        at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)

===========================

Is there anyone experience similar issue ?

environments:

OS   --- CentOS 5.4, Linux 2.6.18-164.15.1.el5 SMP x86_64 GNU/Linux
Java --- build 1.6.0_16-b01, Java HotSpot(TM) 64-Bit Server VM (build
14.2-b01, mixed mode)
Cassandra --- 0.6.0
Node configuration --- node A and node B. both nodes use node A as Seed
client --- Java thrift clients pick one node randomly to do read and write.


-- 
Ingram Chen
online share order: http://dinbendon.net
blog: http://www.javaworld.com.tw/roller/page/ingramchen

Mime
View raw message