cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Eineke (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-5748) When flushing, nodes spent almost 100% in AbstractCompositeType.compare
Date Fri, 12 Jul 2013 13:43:48 GMT
Chris Eineke created CASSANDRA-5748:
---------------------------------------

             Summary: When flushing, nodes spent almost 100% in AbstractCompositeType.compare
                 Key: CASSANDRA-5748
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5748
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 1.2.6, 1.2.5
         Environment: Apache Cassandra v1.2.6

4-node cluster, mostly the same hardware

# java -version
java version "1.6.0_37"
Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)


            Reporter: Chris Eineke
            Priority: Critical


We're pretty heavy users of CQL3 and CQL3 collection types. Occasionally, some nodes of the
cluster will become extremely sluggish and the cluster as a whole starts to become unresponsive,
reads will time out, and nodes will drop mutation messages. This happens when nodes flush
Memtables to disk (based on my tail of the system.log on each node).

I'm a curious guy, so I attached jvisualvm (v1.3.3) to the JVMs that were having this problem.
These nodes are spending up to 98% of CPU in org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:78).
I will attach a thread dump.

Thi is causing us quite a headache, because we're unable to figure what would be causing this.
We tried tuning several configuration settings (column cache size, row key cache size), but
the cluster exhibits the same issues even with the default configuration (except for a modified
num_tokens and listen_address).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message