Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 88259 invoked from network); 18 Apr 2010 01:47:50 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 18 Apr 2010 01:47:50 -0000 Received: (qmail 4092 invoked by uid 500); 18 Apr 2010 01:47:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 4061 invoked by uid 500); 18 Apr 2010 01:47:48 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 4053 invoked by uid 99); 18 Apr 2010 01:47:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Apr 2010 01:47:48 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ingramchen@gmail.com designates 209.85.222.181 as permitted sender) Received: from [209.85.222.181] (HELO mail-pz0-f181.google.com) (209.85.222.181) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Apr 2010 01:47:43 +0000 Received: by pzk11 with SMTP id 11so4729pzk.28 for ; Sat, 17 Apr 2010 18:47:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:content-type; bh=Fth3+NIEsRt/jchiK7gvRIoW9PrKrgV7UFpDKWna6yI=; b=aw1CFPFwAs9l7tOtxnc/zXnRdYgcjsTd0do2I9UV9/aLA6Ji49q43r3mzSjvGaydCD /zO/S0XzELxmar07Mrfd1i+F4pHMDLU32o4rBGvfKXBqxhe2A2UMuUX+mCPA7ibmixJV G+TJacCqhKuHUNKIP1+CDmWD0nhWvbdGDe9W0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=nGfRSluNbRP8GnGWPwfpi+/C97FTOCwbIOrAY2BvvNA8PqNgvmJEonL+i5H2kw5bIv TVCyCjVnd4l8ql/Qn7hjnIYySesgVg/uePPmLQTT+VeURawkmxDPGDFCjLUdAuCJJcVe DhXqju+vODxvYN4c3MY75rZwUeYO/DELav+wY= MIME-Version: 1.0 Received: by 10.140.173.20 with HTTP; Sat, 17 Apr 2010 18:47:23 -0700 (PDT) In-Reply-To: References: Date: Sun, 18 Apr 2010 09:47:23 +0800 Received: by 10.140.58.11 with SMTP id g11mr2179097rva.240.1271555243617; Sat, 17 Apr 2010 18:47:23 -0700 (PDT) Message-ID: Subject: Re: busy thread on IncomingStreamReader ? From: Ingram Chen To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001636b2ac339bacf704847906ea --001636b2ac339bacf704847906ea Content-Type: text/plain; charset=UTF-8 FYI. after upgrading jdk from 1.6.0_16 to 1.6.0_20, the problem solved. On Fri, Apr 16, 2010 at 00:33, Ingram Chen wrote: > Hi all, > > We setup two nodes and simply set replication factor=2 for test run. > > After both nodes, say, node A and node B, serve several hours, we found > that "node A" always keep 300% cpu usage. > (the other node is under 100% cpu, which is normal) > > thread dump on "node A" shows that there are 3 busy threads related to > IncomingStreamReader: > > ========================== > > "Thread-66" prio=10 tid=0x00002aade4018800 nid=0x69e7 runnable > [0x000000004030a000] > java.lang.Thread.State: RUNNABLE > at sun.misc.Unsafe.setMemory(Native Method) > at sun.nio.ch.Util.erase(Util.java:202) > at > sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560) > at > sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) > > "Thread-65" prio=10 tid=0x00002aade4017000 nid=0x69e6 runnable > [0x000000004d44b000] > java.lang.Thread.State: RUNNABLE > at sun.misc.Unsafe.setMemory(Native Method) > at sun.nio.ch.Util.erase(Util.java:202) > at > sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560) > at > sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) > > "Thread-62" prio=10 tid=0x00002aade4014800 nid=0x4150 runnable > [0x000000004d34a000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.FileChannelImpl.size0(Native Method) > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:309) > - locked <0x00002aaac450dcd0> (a java.lang.Object) > at > sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:597) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) > > =========================== > > Is there anyone experience similar issue ? > > environments: > > OS --- CentOS 5.4, Linux 2.6.18-164.15.1.el5 SMP x86_64 GNU/Linux > Java --- build 1.6.0_16-b01, Java HotSpot(TM) 64-Bit Server VM (build > 14.2-b01, mixed mode) > Cassandra --- 0.6.0 > Node configuration --- node A and node B. both nodes use node A as Seed > client --- Java thrift clients pick one node randomly to do read and write. > > > -- > Ingram Chen > online share order: http://dinbendon.net > blog: http://www.javaworld.com.tw/roller/page/ingramchen > -- Ingram Chen online share order: http://dinbendon.net blog: http://www.javaworld.com.tw/roller/page/ingramchen --001636b2ac339bacf704847906ea Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable FYI.

after upgrading jdk from=C2=A0 1.6.0_16 to=C2=A0 1.6.0_20, the = problem solved.

On Fri, Apr 16, 2010 at 0= 0:33, Ingram Chen <ingramchen@gmail.com> wrote:
Hi all,

=C2=A0We setup two nodes and simply set replication fa= ctor=3D2 for test run.

After both nodes, say, node A and node B, serve se= veral hours, we found that "node A" always keep 300% cpu usage.= =C2=A0
(the other node is under 100% cpu, which is normal)
thread dump on "node A" shows that there are 3 = busy threads related to IncomingStreamReader:

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D

"Thread-66" prio=3D10 tid=3D0x00002a= ade4018800 nid=3D0x69e7 runnable [0x000000004030a000]
=C2=A0=C2= =A0 java.lang.Thread.State: RUNNABLE
=C2=A0=C2=A0 =C2=A0 =C2=A0 = =C2=A0at sun.misc.Unsafe.setMemory(Native Method)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.Util.erase(Util.java:20= 2)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl= .transferFromArbitraryChannel(FileChannelImpl.java:560)
=C2=A0=C2= =A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.transferFrom(FileChan= nelImpl.java:603)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.streaming.Inc= omingStreamReader.read(IncomingStreamReader.java:62)
=C2=A0=C2=A0= =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.net.IncomingTcpConnection.run(= IncomingTcpConnection.java:66)

"Thread-65" prio=3D10 tid=3D0x00002aade4017000 nid=3D0x69e6 = runnable [0x000000004d44b000]
=C2=A0=C2=A0 java.lang.Thread.State= : RUNNABLE
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.misc.Unsafe.se= tMemory(Native Method)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.ni= o.ch.Util.erase(Util.java:202)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.transfe= rFromArbitraryChannel(FileChannelImpl.java:560)
=C2=A0=C2=A0 =C2= =A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl= .java:603)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassand= ra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.net.IncomingT= cpConnection.run(IncomingTcpConnection.java:66)

&q= uot;Thread-62" prio=3D10 tid=3D0x00002aade4014800 nid=3D0x4150 runnabl= e [0x000000004d34a000]
=C2=A0=C2=A0 java.lang.Thread.State: RUNNABLE
=C2=A0=C2=A0 = =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.size0(Native Method)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.size(F= ileChannelImpl.java:309)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0- locke= d <0x00002aaac450dcd0> (a java.lang.Object)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannelImpl.transfe= rFrom(FileChannelImpl.java:597)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0= at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamR= eader.java:62)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cas= sandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D

Is there anyone experi= ence similar issue ?

environments:=C2=A0

OS =C2=A0 --- CentOS 5.4, Linux 2.6.18-164.15.1.= el5 SMP x86_64 GNU/Linux
Java --- build 1.6.0_16-b01, Java HotSpot(TM) 64-Bit Server VM (build = 14.2-b01, mixed mode)
Cassandra --- 0.6.0
Node configur= ation --- node A and node B. both nodes use node A as Seed
client= ---=C2=A0Java thrift clients=C2=A0pick one node=C2=A0randomly=C2=A0to do r= ead and write.


--
Ingram C= hen
online share order: http://dinbendon.net
blog: http://www.javaworld.com.tw/rol= ler/page/ingramchen



--
Ingram Chen
o= nline share order: http://dinbendon.net
blog: = http://www.javaworld.com.tw/roller/page/ingramchen
--001636b2ac339bacf704847906ea--