cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Rosello <orose...@corp.free.fr>
Subject High CPU usage on all nodes without any read or write
Date Wed, 07 Jul 2010 10:05:45 GMT
Hi,

We are testing Cassandra here, we would like to use it to store some data :
- about 1000 inserts / seconds in a CF "RAW" :
   Column : TimeUUID (timeuuid of the insert, so 1000 new columns / second)
   Row : YYYYMMDDHH of the insert (to minimize the size of rows, the biggest one is 2GB data),
for example for today at 1:30pm, the row of an insert will be 2010070713
- 4 CF we are using for indexes, all have the same structure
   Column : TimeUUID of the insert (about 300 new columns per second, each correspond to the
same timeuuid of the CF "RAW".
   Row : a String (Customer Id).

The goal is for a Customer, to be able to get all its "RAW" for a time range.

Currently Cassandra is running on 4 nodes / 16 GB RAM each, 8 GB JVM_MAX_MEM

It runs correctly during several days. Last night, we started to have timeout exception on
insert and high cpu load on all nodes.

We stopped inserts. But the CPU remains high (without any insert or read).

We tried to stop and start nodes again, cpu remains high (between 100% and 150%).


So currently, we can't add more data (timeout exceptions, due to high cpu usage I guess).

Could you help me to understand what's wrong ? :)

Cheers,

Olivier


Here are some data :

We are using Cassandra 0.6.2

Address       Status     Load          Range                                      Ring
                                       170141183460469231731687303715884105728    
192.168.0.1  Up         23.43 GB      42535295865117307932921825928971026432     |<--|
192.168.0.2  Up         23.12 GB      85070591730234615865843651857942052864     |   |
192.168.0.3  Up         24.47 GB      127605887595351923798765477786913079296    |   |
192.168.0.4  Up         24.14 GB      170141183460469231731687303715884105728    |-->|



Keyspace: system
	Read Count: 8
	Read Latency: 60.38625 ms.
	Write Count: 13
	Write Latency: 0.3583076923076923 ms.
	Pending Tasks: 0
		Column Family: HintsColumnFamily
		SSTable count: 1
		Space used (live): 103812
		Space used (total): 103812
		Memtable Columns Count: 0
		Memtable Data Size: 0
		Memtable Switch Count: 0
		Read Count: 5
		Read Latency: 83,440 ms.
		Write Count: 0
		Write Latency: NaN ms.
		Pending Tasks: 0
		Key cache capacity: 1
		Key cache size: 1
		Key cache hit rate: 0.6666666666666666
		Row cache: disabled
		Compacted row minimum size: 0
		Compacted row maximum size: 0
		Compacted row mean size: 0

		Column Family: LocationInfo
		SSTable count: 3
		Space used (live): 3999
		Space used (total): 3999
		Memtable Columns Count: 7
		Memtable Data Size: 227
		Memtable Switch Count: 2
		Read Count: 3
		Read Latency: 21,964 ms.
		Write Count: 13
		Write Latency: 0,358 ms.
		Pending Tasks: 0
		Key cache capacity: 3
		Key cache size: 3
		Key cache hit rate: 0.2
		Row cache: disabled
		Compacted row minimum size: 0
		Compacted row maximum size: 0
		Compacted row mean size: 0

----------------
Keyspace: MyKeySpace
	Read Count: 5
	Read Latency: 22737.799 ms.
	Write Count: 0
	Write Latency: NaN ms.
	Pending Tasks: 0
		Column Family: INDEX1
		SSTable count: 5
		Space used (live): 1502023165
		Space used (total): 1502023165
		Memtable Columns Count: 0
		Memtable Data Size: 0
		Memtable Switch Count: 0
		Read Count: 1
		Read Latency: 0,876 ms.
		Write Count: 0
		Write Latency: NaN ms.
		Pending Tasks: 0
		Key cache capacity: 200000
		Key cache size: 0
		Key cache hit rate: NaN
		Row cache: disabled
		Compacted row minimum size: 0
		Compacted row maximum size: 0
		Compacted row mean size: 0

		Column Family: INDEX2
		SSTable count: 3
		Space used (live): 1262977087
		Space used (total): 1262977087
		Memtable Columns Count: 0
		Memtable Data Size: 0
		Memtable Switch Count: 0
		Read Count: 1
		Read Latency: 0,547 ms.
		Write Count: 0
		Write Latency: NaN ms.
		Pending Tasks: 0
		Key cache capacity: 200000
		Key cache size: 0
		Key cache hit rate: NaN
		Row cache: disabled
		Compacted row minimum size: 0
		Compacted row maximum size: 0
		Compacted row mean size: 0

		Column Family: RAW
		SSTable count: 6
		Space used (live): 18867913004
		Space used (total): 18867913004
		Memtable Columns Count: 0
		Memtable Data Size: 0
		Memtable Switch Count: 0
		Read Count: 1
		Read Latency: 113685,183 ms.
		Write Count: 0
		Write Latency: NaN ms.
		Pending Tasks: 0
		Key cache capacity: 200000
		Key cache size: 3
		Key cache hit rate: 0.0
		Row cache: disabled
		Compacted row minimum size: 215734
		Compacted row maximum size: 1055255881
		Compacted row mean size: 491170239

		Column Family: INDEX3
		SSTable count: 6
		Space used (live): 1667728231
		Space used (total): 1667728231
		Memtable Columns Count: 0
		Memtable Data Size: 0
		Memtable Switch Count: 0
		Read Count: 1
		Read Latency: 1,241 ms.
		Write Count: 0
		Write Latency: NaN ms.
		Pending Tasks: 0
		Key cache capacity: 200000
		Key cache size: 0
		Key cache hit rate: NaN
		Row cache: disabled
		Compacted row minimum size: 0
		Compacted row maximum size: 0
		Compacted row mean size: 0

		Column Family: INDEX4
		SSTable count: 7
		Space used (live): 2620328669
		Space used (total): 2620328669
		Memtable Columns Count: 0
		Memtable Data Size: 0
		Memtable Switch Count: 0
		Read Count: 1
		Read Latency: 1,148 ms.
		Write Count: 0
		Write Latency: NaN ms.
		Pending Tasks: 0
		Key cache capacity: 200000
		Key cache size: 0
		Key cache hit rate: NaN
		Row cache: disabled
		Compacted row minimum size: 0
		Compacted row maximum size: 0
		Compacted row mean size: 0

Mime
View raw message