Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 78894 invoked from network); 15 Apr 2010 16:34:22 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 15 Apr 2010 16:34:22 -0000 Received: (qmail 90066 invoked by uid 500); 15 Apr 2010 16:34:21 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 90044 invoked by uid 500); 15 Apr 2010 16:34:21 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 90035 invoked by uid 99); 15 Apr 2010 16:34:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:34:21 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ingramchen@gmail.com designates 209.85.219.225 as permitted sender) Received: from [209.85.219.225] (HELO mail-ew0-f225.google.com) (209.85.219.225) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:34:16 +0000 Received: by ewy25 with SMTP id 25so671025ewy.27 for ; Thu, 15 Apr 2010 09:33:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:received:message-id :subject:from:to:content-type; bh=NAo5oF6T3rrHgNv6DwhampXme893rAd3Uv75hx5AXSc=; b=UmPTeyMjWJkNaTuWOtRzfsj4cpzSaga/HC3F6URFl+AUaWW2G71Eam6DAC3334oNRD e3h7SxDxyFV4PH2sZ4ZqJDhxQ/8Itbg7hFS5t1VOM3yNyK6nVvumayu+TQya92SR4ZEq ESRlQvCAsEVzFxEGSGt7FBW4qAlmoL5LFrpuA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=V+zstchOJ+1EC1QRHalLwooPPYO1XsWdCdxsBRjYstr+QbAFxcREcGZzR6xFxFZJyq omMMiZ0nH6cadb+rTJGdUuY8lWy8pI6dX1Dh+LC8MFSmthu5XIg/C1SVrfIoribYXla8 XCcy/9WszTWWzyEpXYW+4fVAkcqX9RnXG5Q4w= MIME-Version: 1.0 Received: by 10.213.26.81 with HTTP; Thu, 15 Apr 2010 09:33:54 -0700 (PDT) Date: Fri, 16 Apr 2010 00:33:54 +0800 Received: by 10.213.25.78 with SMTP id y14mr180822ebb.75.1271349234952; Thu, 15 Apr 2010 09:33:54 -0700 (PDT) Message-ID: Subject: busy thread on IncomingStreamReader ? From: Ingram Chen To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=00151744790e88ea3f0484490fcb --00151744790e88ea3f0484490fcb Content-Type: text/plain; charset=UTF-8 Hi all, We setup two nodes and simply set replication factor=2 for test run. After both nodes, say, node A and node B, serve several hours, we found that "node A" always keep 300% cpu usage. (the other node is under 100% cpu, which is normal) thread dump on "node A" shows that there are 3 busy threads related to IncomingStreamReader: ========================== "Thread-66" prio=10 tid=0x00002aade4018800 nid=0x69e7 runnable [0x000000004030a000] java.lang.Thread.State: RUNNABLE at sun.misc.Unsafe.setMemory(Native Method) at sun.nio.ch.Util.erase(Util.java:202) at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560) at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) "Thread-65" prio=10 tid=0x00002aade4017000 nid=0x69e6 runnable [0x000000004d44b000] java.lang.Thread.State: RUNNABLE at sun.misc.Unsafe.setMemory(Native Method) at sun.nio.ch.Util.erase(Util.java:202) at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560) at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) "Thread-62" prio=10 tid=0x00002aade4014800 nid=0x4150 runnable [0x000000004d34a000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileChannelImpl.size0(Native Method) at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:309) - locked <0x00002aaac450dcd0> (a java.lang.Object) at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:597) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) =========================== Is there anyone experience similar issue ? environments: OS --- CentOS 5.4, Linux 2.6.18-164.15.1.el5 SMP x86_64 GNU/Linux Java --- build 1.6.0_16-b01, Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode) Cassandra --- 0.6.0 Node configuration --- node A and node B. both nodes use node A as Seed client --- Java thrift clients pick one node randomly to do read and write. -- Ingram Chen online share order: http://dinbendon.net blog: http://www.javaworld.com.tw/roller/page/ingramchen --00151744790e88ea3f0484490fcb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi all,

=C2=A0We setup two nodes and simply s= et replication factor=3D2 for test run.

After= both nodes, say, node A and node B, serve several hours, we found that &qu= ot;node A" always keep 300% cpu usage.=C2=A0
(the other node is under 100% cpu, which is normal)
thread dump on "node A" shows that there are 3 = busy threads related to IncomingStreamReader:

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D

"Thread-66" prio=3D10 tid=3D0x00002a= ade4018800 nid=3D0x69e7 runnable [0x000000004030a000]
=C2=A0=C2= =A0 java.lang.Thread.State: RUNNABLE
=C2=A0=C2=A0 =C2=A0 =C2=A0 = =C2=A0at sun.misc.Unsafe.setMemory(Native Method)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.Util.erase(Util.java:20= 2)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl= .transferFromArbitraryChannel(FileChannelImpl.java:560)
=C2=A0=C2= =A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.transferFrom(FileChan= nelImpl.java:603)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.streaming.Inc= omingStreamReader.read(IncomingStreamReader.java:62)
=C2=A0=C2=A0= =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.net.IncomingTcpConnection.run(= IncomingTcpConnection.java:66)

"Thread-65" prio=3D10 tid=3D0x00002aade4017000 nid=3D0x69e6 = runnable [0x000000004d44b000]
=C2=A0=C2=A0 java.lang.Thread.State= : RUNNABLE
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.misc.Unsafe.se= tMemory(Native Method)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.ni= o.ch.Util.erase(Util.java:202)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.transfe= rFromArbitraryChannel(FileChannelImpl.java:560)
=C2=A0=C2=A0 =C2= =A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl= .java:603)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassand= ra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.net.IncomingT= cpConnection.run(IncomingTcpConnection.java:66)

&q= uot;Thread-62" prio=3D10 tid=3D0x00002aade4014800 nid=3D0x4150 runnabl= e [0x000000004d34a000]
=C2=A0=C2=A0 java.lang.Thread.State: RUNNABLE
=C2=A0=C2=A0 = =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.size0(Native Method)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.size(F= ileChannelImpl.java:309)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0- locke= d <0x00002aaac450dcd0> (a java.lang.Object)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.transfe= rFrom(FileChannelImpl.java:597)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0= at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamR= eader.java:62)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cas= sandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D

Is there anyone experi= ence similar issue ?

environments:=C2=A0

OS =C2=A0 --- CentOS 5.4, Linux 2.6.18-164.15.1.= el5 SMP x86_64 GNU/Linux
Java --- build 1.6.0_16-b01, Java HotSpot(TM) 64-Bit Server VM (build = 14.2-b01, mixed mode)
Cassandra --- 0.6.0
Node configur= ation --- node A and node B. both nodes use node A as Seed
client= ---=C2=A0Java thrift clients=C2=A0pick one node=C2=A0randomly=C2=A0to do r= ead and write.


--
Ingram Chen
online share orde= r: http://dinbendon.net
blog: http://www.javaw= orld.com.tw/roller/page/ingramchen
--00151744790e88ea3f0484490fcb--