cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua McKenzie (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-9923) stress write and counter_write hangs
Date Fri, 18 Dec 2015 16:38:46 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Joshua McKenzie updated CASSANDRA-9923:
---------------------------------------
    Assignee: T Jake Luciani

> stress write and counter_write hangs
> ------------------------------------
>
>                 Key: CASSANDRA-9923
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9923
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Robert Stupp
>            Assignee: T Jake Luciani
>              Labels: stress
>
> (Sorry for the vague description)
> I tried some cstar tests against counter columns. But all these tests against 2.1 and
2.2 ended (hang) during with the following output:
> {noformat}
> Created keyspaces. Sleeping 3s for propagation.
> Sleeping 2s...
> Warming up COUNTER_WRITE with 150000 iterations...
> Running COUNTER_WRITE with 300 threads for 15000000 iteration
> type,      total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,
   .999,     max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb
> total,         98073,   98054,   98054,   98054,     3.1,     1.7,     8.9,    23.2,
   89.9,   107.7,    1.0,  0.00000,      0,      0,       0,       0,       0,       0
> total,        188586,   72492,   72492,   72492,     4.1,     1.5,    10.0,    61.4,
  202.8,   214.7,    2.2,  0.13101,      0,      3,     564,     564,       6,    3447
> total,        363880,  137986,  137986,  137986,     2.2,     1.4,     4.1,     9.6,
  207.1,   253.3,    3.5,  0.18684,      0,      0,       0,       0,       0,       0
> total,        460122,  105062,  105062,  105062,     2.8,     1.4,     4.6,    14.7,
  225.6,   236.2,    4.4,  0.13969,      0,      1,     214,     214,       0,    1199
> total,        600625,  111453,  111453,  111453,     2.7,     1.4,     3.8,    10.4,
  231.5,   241.6,    5.7,  0.11366,      0,      2,     442,     442,       1,    2389
> total,        745680,  149583,  149583,  149583,     2.0,     1.4,     3.6,     6.7,
  155.8,   159.7,    6.7,  0.11318,      0,      0,       0,       0,       0,       0
> total,        828453,   63632,   63632,   63632,     4.7,     1.4,     4.8,   261.9,
  274.5,   279.3,    8.0,  0.12645,      0,      3,     782,     782,       1,    3542
> total,       1009560,  172429,  172429,  172429,     1.7,     1.4,     3.7,     6.1,
   16.2,    29.7,    9.0,  0.11629,      0,      0,       0,       0,       0,       0
> total,       1062409,   53860,   53860,   53860,     5.5,     1.3,     8.6,   270.3,
  293.4,   324.3,   10.0,  0.12738,      0,      2,     542,     542,       7,    2354
> total,       1186672,   96540,   96540,   96540,     3.1,     1.5,     5.9,    14.5,
  266.4,   277.6,   11.3,  0.11451,      0,      1,     260,     260,       0,    1183
> {noformat}
> ...
> {noformat}
> total,       4977251,     238,     238,     238,     0.7,     0.6,     0.7,     1.3,
    3.4,   158.5,  352.3,  0.11749,      0,      0,       0,       0,       0,       0
> total,       4979839,     214,     214,     214,     0.6,     0.6,     0.7,     1.3,
    2.5,     2.8,  364.4,  0.11761,      0,      0,       0,       0,       0,       0
> total,       4981729,     191,     191,     191,     0.6,     0.6,     0.7,     1.3,
    3.2,     3.3,  374.3,  0.11774,      0,      0,       0,       0,       0,       0
> total,       4983362,     167,     167,     167,     0.8,     0.7,     1.8,     2.7,
    3.9,     5.8,  384.0,  0.11787,      0,      0,       0,       0,       0,       0
> total,       4985171,     153,     153,     153,     0.7,     0.6,     1.2,     1.4,
    2.0,     3.3,  395.9,  0.11799,      0,      0,       0,       0,       0,       0
> total,       4986684,     137,     137,     137,     0.7,     0.6,     0.8,     1.3,
    2.0,     2.0,  406.9,  0.11812,      0,      0,       0,       0,       0,       0
> total,       4988410,     121,     121,     121,     0.7,     0.7,     0.8,     1.3,
    2.0,     2.8,  421.1,  0.11824,      0,      0,       0,       0,       0,       0
> total,       4990216,      99,      99,      99,     0.7,     0.7,     0.8,     1.4,
    2.6,     2.8,  439.5,  0.11836,      0,      0,       0,       0,       0,       0
> total,       4991765,      81,      81,      81,     0.8,     0.7,     0.8,     1.4,
   30.1,    81.6,  458.7,  0.11848,      0,      1,     159,     159,       0,    1179
> total,       4993731,      67,      67,      67,     0.7,     0.7,     0.8,     1.4,
    3.2,     3.2,  488.1,  0.11860,      0,      0,       0,       0,       0,       0
> total,       4996565,      45,      45,      45,     0.9,     0.7,     0.9,     1.5,
   84.7,   218.3,  551.5,  0.11872,      0,      1,     248,     248,       0,    1180
> java.lang.RuntimeException: Timed out waiting for a timer thread - seems one got stuck.
Check GC/Heap size
> 	at org.apache.cassandra.stress.util.Timing.snap(Timing.java:98)
> 	at org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156)
> 	at org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:37)
> 	at org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I don't know whether it's a stress or a counter issue - but wanted to record it for further
investigation.
> [Console log file|http://cstar.datastax.com/tests/artifacts/6cd54260-35d6-11e5-a6ff-42010af0688f/console]
> cstar test steps:
> # counter_write n=15M -pop dist=gaussian\(1..5M\) -rate threads=300
> # counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=300
> # counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=50



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message