Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4919A10F81 for ; Fri, 12 Jul 2013 13:49:55 +0000 (UTC) Received: (qmail 18100 invoked by uid 500); 12 Jul 2013 13:49:52 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 17885 invoked by uid 500); 12 Jul 2013 13:49:50 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 17798 invoked by uid 99); 12 Jul 2013 13:49:49 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Jul 2013 13:49:49 +0000 Date: Fri, 12 Jul 2013 13:49:49 +0000 (UTC) From: "Sylvain Lebresne (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (CASSANDRA-5748) When flushing, nodes spent almost 100% in AbstractCompositeType.compare MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne resolved CASSANDRA-5748. ----------------------------------------- Resolution: Duplicate This is a duplicate of CASSANDRA-5677. It will get fixed there. > When flushing, nodes spent almost 100% in AbstractCompositeType.compare > ----------------------------------------------------------------------- > > Key: CASSANDRA-5748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5748 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 1.2.5, 1.2.6 > Environment: Apache Cassandra v1.2.6 > 4-node cluster, mostly the same hardware > # java -version > java version "1.6.0_37" > Java(TM) SE Runtime Environment (build 1.6.0_37-b06) > Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) > Reporter: Chris Eineke > Priority: Critical > Attachments: thread_dump > > > We're pretty heavy users of CQL3 and CQL3 collection types. Occasionally, some nodes of the cluster will become extremely sluggish and the cluster as a whole starts to become unresponsive, reads will time out, and nodes will drop mutation messages. This happens when nodes flush Memtables to disk (based on my tail of the system.log on each node). > I'm a curious guy, so I attached jvisualvm (v1.3.3) to the JVMs that were having this problem. These nodes are spending up to 98% of CPU in org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:78). I will attach a thread dump. > Thi is causing us quite a headache, because we're unable to figure what would be causing this. We tried tuning several configuration settings (column cache size, row key cache size), but the cluster exhibits the same issues even with the default configuration (except for a modified num_tokens and listen_address). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira