From user-return-6407-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Fri Jun 04 04:56:27 2010 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 29555 invoked from network); 4 Jun 2010 04:56:27 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 4 Jun 2010 04:56:27 -0000 Received: (qmail 5912 invoked by uid 500); 4 Jun 2010 04:56:26 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 5870 invoked by uid 500); 4 Jun 2010 04:56:26 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 5862 invoked by uid 500); 4 Jun 2010 04:56:26 -0000 Delivered-To: apmail-incubator-cassandra-user@incubator.apache.org Received: (qmail 5858 invoked by uid 99); 4 Jun 2010 04:56:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:56:26 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of xluke@live.com designates 65.55.116.96 as permitted sender) Received: from [65.55.116.96] (HELO blu0-omc3-s21.blu0.hotmail.com) (65.55.116.96) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:56:16 +0000 Received: from BLU123-DS6 ([65.55.116.72]) by blu0-omc3-s21.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 3 Jun 2010 21:55:55 -0700 X-Originating-IP: [119.145.71.82] X-Originating-Email: [xluke@live.com] Message-ID: From: "Lu Ming" To: References: <139EF7B8AE318D408CFCB51C899CAF3403C8D5E9F3@cpexch01.iovation.com> <4B968EB7.7050407@real.com> In-Reply-To: Subject: High CPU Usage since 0.6.2 Date: Fri, 4 Jun 2010 12:55:53 +0800 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal Importance: Normal X-Mailer: Microsoft Windows Live Mail 14.0.8089.726 X-MimeOLE: Produced By Microsoft MimeOLE V14.0.8089.726 X-OriginalArrivalTime: 04 Jun 2010 04:55:55.0349 (UTC) FILETIME=[370EC050:01CB03A2] X-Virus-Checked: Checked by ClamAV on apache.org I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to cassandra to 0.6.2 yesterday. But today I find six cassandra nodes have high CPU usage more than 400% in my 8-core CPU sever. The worst one is more than 760%. It is very serious. I use jvisualvm to watch the worst node, and I found that there are many running threads named "thread-xxx" the status of other threads is waiting and sleeping. "Thread-130" - Thread t@240 java.lang.Thread.State: RUNNABLE at sun.misc.Unsafe.setMemory(Native Method) at sun.nio.ch.Util.erase(Util.java:202) at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560) at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) Locked ownable synchronizers: - None "Thread-126" - Thread t@236 java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:200) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) - locked java.lang.Object@10808561 at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565) at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) Locked ownable synchronizers: - None "Thread-119" - Thread t@229 java.lang.Thread.State: RUNNABLE at sun.nio.ch.NativeThread.current(Native Method) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182) - locked java.lang.Object@65b4abbd - locked java.lang.Object@38773975 at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565) at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) Locked ownable synchronizers: - None