cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dikang Gu (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (CASSANDRA-8681) cassandra-stress fails after single ReadTimeoutException
Date Wed, 08 Mar 2017 22:25:38 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-8681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dikang Gu reopened CASSANDRA-8681:
----------------------------------
    Reproduced In: 3.9, 2.2.0 beta 1  (was: 2.2.0 beta 1)

I want to re-open this issue, because I constantly hit this when benchmarking 3.9 branch.

command I use:
{code}
/tools/bin/cassandra-stress  user n=1000000000   profile=/root/cassandra_stress.yaml ops\(insert=2,singlepost=1\)
cl=ONE -node 2401:db00:21:2121:face:0:13:0  -rate threads=100 -log file=~/cass-stress-1b.log
{code}


error:

{code}
total,     123245370,  137904,  119461,  119461,     0.7,     0.5,     1.9,     3.5,     6.6,
   21.4,  980.2,  0.13322,      0,      0,       0,       0,       0,       0
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write
query at consistency ONE (1 replica were required but only 0 acknowledged the write)
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write
query at consistency ONE (1 replica were required but only 0 acknowledged the write)
insert,     82257602,   79373,   79373,   79373,     0.7,     0.4,     1.7,     3.3,    18.2,
  133.2,  981.3,  0.13313,      0,      0,       0,       0,       0,       0
singlepost,  41114666,   40220,   24092,   24092,     1.0,     0.7,     2.1,     4.0,   129.1,
  131.9,  981.3,  0.13313,      0,      0,       0,       0,       0,       0
total,     123372268,  119590,  103462,  103462,     0.8,     0.5,     1.9,     3.6,    18.7,
  133.2,  981.3,  0.13313,      0,      0,       0,       0,       0,       0
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write
query at consistency ONE (1 replica were required but only 0 acknowledged the write)
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write
query at consistency ONE (1 replica were required but only 0 acknowledged the write)
com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query
at consistency ONE (1 responses were required but only 0 replica responded, 1 failed)
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.io.IOException: Operation x10 on key(s) [|=VO
                                                  oq3# H"gz`_U%A'qqp3uH{C(ux-1^6>9=PYq]M:+]:
Error executing: (NoSuchElementException)

        at org.apache.cassandra.stress.Operation.error(Operation.java:138)
        at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
        at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:156)
        at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:434)
FAILURE
java.lang.RuntimeException: Failed to execute stress action
        at org.apache.cassandra.stress.StressAction.run(StressAction.java:86)
        at org.apache.cassandra.stress.Stress.run(Stress.java:133)
        at org.apache.cassandra.stress.Stress.main(Stress.java:61)
[root@cassandrarocksdbtest1-prn <lxc> /data/apache-cassandra-3.9 ] #
{code}



> cassandra-stress fails after single ReadTimeoutException
> --------------------------------------------------------
>
>                 Key: CASSANDRA-8681
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8681
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>            Reporter: Robert Stupp
>            Assignee: T Jake Luciani
>              Labels: stress
>
> After a single exception is propagated from the Java Driver, cassandra-stress prints
out some {{NoSuchElementException}}s and stops working.
> {noformat}
> Running [singlepost, timeline] with 36 threads 2 minutes
> total ops , adj row/s,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,
   .999,     max,   time,   stderr,  gc: #,  max ms,  sum ms,  sdv ms,      mb
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (1 responses were required but only 0 replica responded)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (1 responses were required but only 0 replica responded)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (timeout while waiting for repair of inconsistent replica)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (timeout while waiting for repair of inconsistent replica)
> java.util.NoSuchElementException
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (1 responses were required but only 0 replica responded)
> java.util.NoSuchElementException
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (1 responses were required but only 0 replica responded)
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (1 responses were required but only 0 replica responded)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (1 responses were required but only 0 replica responded)
> java.util.NoSuchElementException
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (1 responses were required but only 0 replica responded)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (1 responses were required but only 0 replica responded)
> java.util.NoSuchElementException
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (1 responses were required but only 0 replica responded)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read
query at consistency ONE (timeout while waiting for repair of inconsistent replica)
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.io.IOException: Operation x10 on key(s) [Z$Vq"zj\Aw}KG2>9$TY(!">e)
>                                                                        m#$D1x??]D?2S]:
Error executing: (NoSuchElementException)
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> java.util.NoSuchElementException
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)java.util.NoSuchElementException
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)java.util.NoSuchElementException
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [ot^XJz*C{
> HYUg+"'	j^36`
>              KX	aj0n;]: Error executing: (NoSuchElementException)
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [\!d,*&M	D4 *^%=c MPZBZ+%W=2J
>                                                                             Ic~s1/[Ba?3}V]:
Error executing: (NoSuchElementException)
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> f\llVm8IMYExceptX"RCSD}:Hx~0-n x&Q on key(s) [3+`&BP0yu,sdQ!s+CYw]ILLPk v)^IJ<z`>SU-
>                                   `]: Error executing: (NoSuchElementException)
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [KDIWY$$D?.+(y2iT="toY
> 5\.^
> _dV]: Error executing: (NoSuchElementException)
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> [(>V 1]: Error executing: (NoSuchElementException)4f}n-K|.ly6q&5-/QLgu~2V{fmw5OJ;+!(v'
<T?#A$
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [$:YcLz5^?(v$aQR7b~JEjd,|]X8R><6u@P(zCTS"d0Z]:
Error executing: (NoSuchElementException)
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [6-1YKka#HB2^o.g{9Mvq({	I9-I^!5S?jhNA>h#A434]:
Error executing: (NoSuchElementException)
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [/aW4D-aS
>                                                       lv2)fa_0I+4$
>                                                                   C&]}WedM^B9D<v.\]:
Error executing: (NoSuchElementException)
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [lk/!
>                                                   py%
> u"@$%wO4)2q]#j+1kE+0
> x.8lvA}"R6}f_]: Error executing: (NoSuchElementException)
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
>  $ e}*#D*T&*j Ction: Operation x10 on key(s) [
>                T6{DlsMm'	u.z]: Error executing: (NoSuchElementException)
> 	at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> 	at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> 	at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> 	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message