cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shawn Kumar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-8008) "Timed out waiting for timer thread" on large stress command
Date Fri, 26 Sep 2014 20:08:34 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-8008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shawn Kumar updated CASSANDRA-8008:
-----------------------------------
    Description: 
I've been using cstar_perf to test a performance scenario and was able to reproduce this error
on stock 2.1.0 while carrying out large stress writes (50M keys):
{noformat}
java.lang.RuntimeException: Timed out waiting for a timer thread - seems one got stuck
        at org.apache.cassandra.stress.util.Timing.snap(Timing.java:83)
        at org.apache.cassandra.stress.util.Timing.snap(Timing.java:118)
        at org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156)
        at org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:42)
        at org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104)
        at java.lang.Thread.run(Thread.java:745)

{noformat}
It looks similar to CASSANDRA-6943, but that should have fixed it, and I haven't been able
to consistently replicate this with other runs. This particular run was stress writing/reading
about 300M keys, and is an early attempt at carrying out a test of this size so perhaps it
only manifests with larger tests. 

  was:
I've been using cstar_perf to test cassandra with different gc's and came across this error
on one run which effectively stopped the test:

java.lang.RuntimeException: Timed out waiting for a timer thread - seems one got stuck at
org.apache.cassandra.stress.util.Timing.snap(Timing.java:83)

It looks similar to CASSANDRA-6943, but that should have fixed it, and I haven't been able
to consistently replicate this with other runs. This particular run was stress writing/reading
about 300M keys, and is an early attempt at carrying out a test of this size so perhaps it
only manifests with larger tests. 

The modifications from stock 2.1.0 were changes to heap size and usage of g1gc, as well as
using offheap_objects. I have attached thread dumps from the nodes in question, hopefully
they capture the broken state. I am continuing to test this, and will see if I can reproduce
this again.


> "Timed out waiting for timer thread" on large stress command
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-8008
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8008
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Shawn Kumar
>         Attachments: node1.log, node2.log
>
>
> I've been using cstar_perf to test a performance scenario and was able to reproduce this
error on stock 2.1.0 while carrying out large stress writes (50M keys):
> {noformat}
> java.lang.RuntimeException: Timed out waiting for a timer thread - seems one got stuck
>         at org.apache.cassandra.stress.util.Timing.snap(Timing.java:83)
>         at org.apache.cassandra.stress.util.Timing.snap(Timing.java:118)
>         at org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156)
>         at org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:42)
>         at org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}
> It looks similar to CASSANDRA-6943, but that should have fixed it, and I haven't been
able to consistently replicate this with other runs. This particular run was stress writing/reading
about 300M keys, and is an early attempt at carrying out a test of this size so perhaps it
only manifests with larger tests. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message