cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Stepura (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7220) Nodes hang with 100% CPU load
Date Sat, 07 Mar 2015 20:19:38 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351775#comment-14351775
] 

Mikhail Stepura commented on CASSANDRA-7220:
--------------------------------------------

I've seen similar behavior because of CASSANDRA-8485, and there is ~20min delay in flushing
in the logs
{code}
 INFO [OptionalTasks:1] 2014-08-17 23:13:53,303 MeteredFlusher.java (line 58) flushing high-traffic
column family CFS(Keyspace='services', ColumnFamily='service_request_count_per_minute') (estimated
16777216 bytes)
..
 INFO [OptionalTasks:1] 2014-08-17 23:34:26,217 MeteredFlusher.java (line 58) flushing high-traffic
column family CFS(Keyspace='services', ColumnFamily='service_request_count') (estimated 12582912
bytes)
{code}

> Nodes hang with 100% CPU load
> -----------------------------
>
>                 Key: CASSANDRA-7220
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7220
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: C* 2.0.7
> 4 nodes cluster on 12 core machines
>            Reporter: Robert Stupp
>         Attachments: c-12-read-100perc-cpu.zip, system.log
>
>
> I've ran a test that both reads and writes rows.
> After some time, all writes succeeded and all reads stopped.
> Two of the four nodes have 16 of 16 threads of the "ReadStage" thread pool running. The
number of pending task continuouly grows on these nodes.
> I have attached outputs of the stack traces and some diagnostic output from "nodetool
tpstats"
> "nodetool status" shows all nodes as UN.
> I had run that test previously without any issues in with the same configuration.
> Some "specials" from cassandra.yaml:
> - key_cache_size_in_mb: 1024
> - row_cache_size_in_mb: 8192
> The nodes running at 100% CPU are "node2" and "node3". node1&node4 are fine.
> I'm not sure if it is reproducable - but it's definitly not a good behaviour.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message