Hi

If the reason for this happening is compaction,
changing the priority of the compaction thread might be effective. (cassandra0.6.3 offers this function)

see also 0.6.3 changelog

2010/7/7 Olivier Rosello <orosello@corp.free.fr>
Hi,

We are testing Cassandra here, we would like to use it to store some data :
- about 1000 inserts / seconds in a CF "RAW" :
  Column : TimeUUID (timeuuid of the insert, so 1000 new columns / second)
  Row : YYYYMMDDHH of the insert (to minimize the size of rows, the biggest one is 2GB data), for example for today at 1:30pm, the row of an insert will be 2010070713
- 4 CF we are using for indexes, all have the same structure
  Column : TimeUUID of the insert (about 300 new columns per second, each correspond to the same timeuuid of the CF "RAW".
  Row : a String (Customer Id).

The goal is for a Customer, to be able to get all its "RAW" for a time range.

Currently Cassandra is running on 4 nodes / 16 GB RAM each, 8 GB JVM_MAX_MEM

It runs correctly during several days. Last night, we started to have timeout exception on insert and high cpu load on all nodes.

We stopped inserts. But the CPU remains high (without any insert or read).

We tried to stop and start nodes again, cpu remains high (between 100% and 150%).


So currently, we can't add more data (timeout exceptions, due to high cpu usage I guess).

Could you help me to understand what's wrong ? :)

Cheers,

Olivier


Here are some data :

We are using Cassandra 0.6.2

Address       Status     Load          Range                                      Ring
                                      170141183460469231731687303715884105728
192.168.0.1  Up         23.43 GB      42535295865117307932921825928971026432     |<--|
192.168.0.2  Up         23.12 GB      85070591730234615865843651857942052864     |   |
192.168.0.3  Up         24.47 GB      127605887595351923798765477786913079296    |   |
192.168.0.4  Up         24.14 GB      170141183460469231731687303715884105728    |-->|



Keyspace: system
       Read Count: 8
       Read Latency: 60.38625 ms.
       Write Count: 13
       Write Latency: 0.3583076923076923 ms.
       Pending Tasks: 0
               Column Family: HintsColumnFamily
               SSTable count: 1
               Space used (live): 103812
               Space used (total): 103812
               Memtable Columns Count: 0
               Memtable Data Size: 0
               Memtable Switch Count: 0
               Read Count: 5
               Read Latency: 83,440 ms.
               Write Count: 0
               Write Latency: NaN ms.
               Pending Tasks: 0
               Key cache capacity: 1
               Key cache size: 1
               Key cache hit rate: 0.6666666666666666
               Row cache: disabled
               Compacted row minimum size: 0
               Compacted row maximum size: 0
               Compacted row mean size: 0

               Column Family: LocationInfo
               SSTable count: 3
               Space used (live): 3999
               Space used (total): 3999
               Memtable Columns Count: 7
               Memtable Data Size: 227
               Memtable Switch Count: 2
               Read Count: 3
               Read Latency: 21,964 ms.
               Write Count: 13
               Write Latency: 0,358 ms.
               Pending Tasks: 0
               Key cache capacity: 3
               Key cache size: 3
               Key cache hit rate: 0.2
               Row cache: disabled
               Compacted row minimum size: 0
               Compacted row maximum size: 0
               Compacted row mean size: 0

----------------
Keyspace: MyKeySpace
       Read Count: 5
       Read Latency: 22737.799 ms.
       Write Count: 0
       Write Latency: NaN ms.
       Pending Tasks: 0
               Column Family: INDEX1
               SSTable count: 5
               Space used (live): 1502023165
               Space used (total): 1502023165
               Memtable Columns Count: 0
               Memtable Data Size: 0
               Memtable Switch Count: 0
               Read Count: 1
               Read Latency: 0,876 ms.
               Write Count: 0
               Write Latency: NaN ms.
               Pending Tasks: 0
               Key cache capacity: 200000
               Key cache size: 0
               Key cache hit rate: NaN
               Row cache: disabled
               Compacted row minimum size: 0
               Compacted row maximum size: 0
               Compacted row mean size: 0

               Column Family: INDEX2
               SSTable count: 3
               Space used (live): 1262977087
               Space used (total): 1262977087
               Memtable Columns Count: 0
               Memtable Data Size: 0
               Memtable Switch Count: 0
               Read Count: 1
               Read Latency: 0,547 ms.
               Write Count: 0
               Write Latency: NaN ms.
               Pending Tasks: 0
               Key cache capacity: 200000
               Key cache size: 0
               Key cache hit rate: NaN
               Row cache: disabled
               Compacted row minimum size: 0
               Compacted row maximum size: 0
               Compacted row mean size: 0

               Column Family: RAW
               SSTable count: 6
               Space used (live): 18867913004
               Space used (total): 18867913004
               Memtable Columns Count: 0
               Memtable Data Size: 0
               Memtable Switch Count: 0
               Read Count: 1
               Read Latency: 113685,183 ms.
               Write Count: 0
               Write Latency: NaN ms.
               Pending Tasks: 0
               Key cache capacity: 200000
               Key cache size: 3
               Key cache hit rate: 0.0
               Row cache: disabled
               Compacted row minimum size: 215734
               Compacted row maximum size: 1055255881
               Compacted row mean size: 491170239

               Column Family: INDEX3
               SSTable count: 6
               Space used (live): 1667728231
               Space used (total): 1667728231
               Memtable Columns Count: 0
               Memtable Data Size: 0
               Memtable Switch Count: 0
               Read Count: 1
               Read Latency: 1,241 ms.
               Write Count: 0
               Write Latency: NaN ms.
               Pending Tasks: 0
               Key cache capacity: 200000
               Key cache size: 0
               Key cache hit rate: NaN
               Row cache: disabled
               Compacted row minimum size: 0
               Compacted row maximum size: 0
               Compacted row mean size: 0

               Column Family: INDEX4
               SSTable count: 7
               Space used (live): 2620328669
               Space used (total): 2620328669
               Memtable Columns Count: 0
               Memtable Data Size: 0
               Memtable Switch Count: 0
               Read Count: 1
               Read Latency: 1,148 ms.
               Write Count: 0
               Write Latency: NaN ms.
               Pending Tasks: 0
               Key cache capacity: 200000
               Key cache size: 0
               Key cache hit rate: NaN
               Row cache: disabled
               Compacted row minimum size: 0
               Compacted row maximum size: 0
               Compacted row mean size: 0