cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Timeout during stress test
Date Mon, 11 Apr 2011 22:55:27 GMT
TimedOutException means the cluster could not perform the request in rpc_timeout time. The
client should retry as the problem may be transitory. 

In this case read performance may have slowed down due to the number of sstables 286. It hard
to tell without knowing what the workload is.

Aaron

On 12 Apr 2011, at 09:56, mcasandra wrote:

> I see this occurring often when all cassandra nodes all of a sudden show CPU
> spike. All reads fail for about 2 mts. GC.log and system.log doesn't reveal
> much.
> 
> Only think I notice is that when I restart nodes there are tons of files
> that gets deleted. cfstats from one of the nodes looks like this:
> 
> nodetool -h `hostname` tpstats
> Pool Name                    Active   Pending      Completed
> ReadStage                        27        27          21491
> RequestResponseStage              0         0         201641
> MutationStage                     0         0         236513
> ReadRepairStage                   0         0           7222
> GossipStage                       0         0          31498
> AntiEntropyStage                  0         0              0
> MigrationStage                    0         0              0
> MemtablePostFlusher               0         0            324
> StreamStage                       0         0              0
> FlushWriter                       0         0            324
> FILEUTILS-DELETE-POOL             0         0           1220
> MiscStage                         0         0              0
> FlushSorter                       0         0              0
> InternalResponseStage             0         0              0
> HintedHandoff                     1         3              9
> 
> --
> 
> ----------------
> Keyspace: StressKeyspace
>        Read Count: 21957
>        Read Latency: 46.91765058978913 ms.
>        Write Count: 222104
>        Write Latency: 0.008302124230090408 ms.
>        Pending Tasks: 0
>                Column Family: StressStandard
>                SSTable count: 286
>                Space used (live): 377916657941
>                Space used (total): 377916657941
>                Memtable Columns Count: 362
>                Memtable Data Size: 164403613
>                Memtable Switch Count: 326
>                Read Count: 21958
>                Read Latency: 631.464 ms.
>                Write Count: 222104
>                Write Latency: 0.007 ms.
>                Pending Tasks: 0
>                Key cache capacity: 1000000
>                Key cache size: 22007
>                Key cache hit rate: 0.002453626459907744
>                Row cache: disabled
>                Compacted row minimum size: 87
>                Compacted row maximum size: 5839588
>                Compacted row mean size: 552698
> 
> ----------------
> 
> 
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Timeout-during-stress-test-tp6262430p6263087.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.


Mime
View raw message