I have a cassandra setup with 7 Node Ring single DC with RF:7 and Read:CL:1.There is live traffic
on these nodes except one -> the traffic is 90% read. There are also writes happening to
all these nodes which typically are user specific data etc.These nodes aNow at times what
is happening is there are 2-3 say nodes getting in a hanged state and eventually the whole
ring behaves so. There state in terms of CPU Usage is very High 17Loads ,network incoming
and outgoing requests packet were very high and there is continuous GC(Major and Minor) pauses
happening and Read Messages dropped simultaneously.Also the SSTables couunt have increased
and decreased for some families during this time. So majorly its the GC and SStable compaction
and Memtable flush happening. We are running cassandra behind solr using Solandra.The max
docs that we have are around 0.5Million.
We keep one node as the main write node where we run sheduled job thats pools data into the
cluster from MySql db.These jobs are run hourly.Recently we have increased the data to somewhere
to the size of 30X . Before the same setup was stable and these jobs used to run in every
5 minutes apart from the similar problem 2-3 times.
GC:Settings
4Gb Heap Xmx,Xms
2Gb Young Geneartion: Xmn
ParNew
CMS
-XX:+UseParNewGC"
-XX:+UseConcMarkSweepGC"
-XX:+CMSParallelRemarkEnabled"
-XX:SurvivorRatio=8"
-XX:MaxTenuringThreshold=1"
-XX:CMSInitiatingOccupancyFraction=75"
-XX:+UseCMSInitiatingOccupancyOnly"
Memtable Threshold is also:0.75 and rest default cassandra settings
Also we dont have in place a connectionpool for thrift.We are using cassandra:0.8.6 with
solr:3.3
Will share the Cfstats shortly or anything else needed for that matter.Can you guys help
me kn this.
==========================================
Keyspace: system
Read Count: 19
Read Latency: 5.6876842105263155 ms.
Write Count: 2375
Write Latency: 0.010477894736842104 ms.
Pending Tasks: 0
Column Family: NodeIdInfo
SSTable count: 0
Space used (live): 0
Space used (total): 0
Number of Keys (estimate): 0
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 1
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
Column Family: HintsColumnFamily
SSTable count: 2
Space used (live): 492064
Space used (total): 492064
Number of Keys (estimate): 256
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 3
Read Count: 10
Read Latency: 5.643 ms.
Write Count: 2372
Write Latency: 0.010 ms.
Pending Tasks: 0
Key cache capacity: 2
Key cache size: 2
Key cache hit rate: 0.42857142857142855
Row cache: disabled
Compacted row minimum size: 219343
Compacted row maximum size: 263210
Compacted row mean size: 263210
Column Family: Schema
SSTable count: 2
Space used (live): 20827
Space used (total): 20827
Number of Keys (estimate): 256
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 3
Read Latency: 4.253 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 2
Key cache size: 2
Key cache hit rate: 0.0
Row cache: disabled
Compacted row minimum size: 104
Compacted row maximum size: 8239
Compacted row mean size: 3314
Column Family: Migrations
SSTable count: 2
Space used (live): 33180
Space used (total): 33180
Number of Keys (estimate): 256
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 2
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 9888
Compacted row maximum size: 17084
Compacted row mean size: 14474
Column Family: IndexInfo
SSTable count: 0
Space used (live): 0
Space used (total): 0
Number of Keys (estimate): 0
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 1
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
Column Family: LocationInfo
SSTable count: 3
Space used (live): 15844
Space used (total): 15844
Number of Keys (estimate): 384
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 2
Read Count: 6
Read Latency: 6.479 ms.
Write Count: 3
Write Latency: 0.004 ms.
Pending Tasks: 0
Key cache capacity: 3
Key cache size: 3
Key cache hit rate: 0.3333333333333333
Row cache: disabled
Compacted row minimum size: 73
Compacted row maximum size: 310
Compacted row mean size: 126
----------------
Keyspace: L
Read Count: 535675
Read Latency: 4.472875136976712 ms.
Write Count: 282216
Write Latency: 0.08156946806701251 ms.
Pending Tasks: 0
Column Family: FC
SSTable count: 6
Space used (live): 444790202
Space used (total): 444790202
Number of Keys (estimate): 768
Memtable Columns Count: 93231
Memtable Data Size: 172790560
Memtable Switch Count: 1
Read Count: 6651
Read Latency: 62.041 ms.
Write Count: 94235
Write Latency: 0.026 ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 104
Compacted row maximum size: 10090808
Compacted row mean size: 4256880
Column Family: Docs
SSTable count: 7
Space used (live): 1487268134
Space used (total): 1487268134
Number of Keys (estimate): 303744
Memtable Columns Count: 98040
Memtable Data Size: 37732086
Memtable Switch Count: 1
Read Count: 125986
Read Latency: 1.868 ms.
Write Count: 11900
Write Latency: 0.520 ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 36
Compacted row maximum size: 51012
Compacted row mean size: 5547
Column Family: SI
SSTable count: 3
Space used (live): 136512555
Space used (total): 136512555
Number of Keys (estimate): 286848
Memtable Columns Count: 5344
Memtable Data Size: 21139811
Memtable Switch Count: 1
Read Count: 27642
Read Latency: 0.880 ms.
Write Count: 5815
Write Latency: 0.093 ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 73
Compacted row maximum size: 30130992
Compacted row mean size: 5022190
Column Family: TL
SSTable count: 4
Space used (live): 314821524
Space used (total): 314821524
Number of Keys (estimate): 512
Memtable Columns Count: 83428
Memtable Data Size: 16069959
Memtable Switch Count: 1
Read Count: 10867
Read Latency: 17.949 ms.
Write Count: 5386
Write Latency: 0.578 ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 12108971
Compacted row maximum size: 186563160
Compacted row mean size: 90195666
Column Family: TI
SSTable count: 6
Space used (live): 1451685937
Space used (total): 1451685937
Number of Keys (estimate): 3910144
Memtable Columns Count: 166006
Memtable Data Size: 184308808
Memtable Switch Count: 1
Read Count: 364529
Read Latency: 4.194 ms.
Write Count: 164880
Write Latency: 0.065 ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 104
Compacted row maximum size: 7007506
Compacted row mean size: 18922
----------------
Keyspace: LH
Read Count: 61353
Read Latency: 0.20730264208759147 ms.
Write Count: 56289
Write Latency: 0.13461228659240704 ms.
Pending Tasks: 0
Column Family: UserPrefrences
SSTable count: 4
Space used (live): 335309522
Space used (total): 335309522
Number of Keys (estimate): 1513600
Memtable Columns Count: 12807
Memtable Data Size: 9144714
Memtable Switch Count: 1
Read Count: 20132
Read Latency: 0.201 ms.
Write Count: 27817
Write Latency: 0.009 ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 61
Compacted row maximum size: 215
Compacted row mean size: 179
Column Family: LastViewedHotels
SSTable count: 3
Space used (live): 152316912
Space used (total): 152316912
Number of Keys (estimate): 767104
Memtable Columns Count: 3059
Memtable Data Size: 2906758
Memtable Switch Count: 1
Read Count: 9077
Read Latency: 0.180 ms.
Write Count: 3585
Write Latency: 0.018 ms.
Pending Tasks: 0
Key cache: disabled
Row cache capacity: 10000
Row cache size: 565
Row cache hit rate: 0.7896882229811611
Compacted row minimum size: 36
Compacted row maximum size: 51012
Compacted row mean size: 144
Column Family: BookedHotels
SSTable count: 3
Space used (live): 7274700
Space used (total): 7274700
Number of Keys (estimate): 39680
Memtable Columns Count: 7
Memtable Data Size: 392
Memtable Switch Count: 1
Read Count: 139
Read Latency: 0.027 ms.
Write Count: 10
Write Latency: 0.008 ms.
Pending Tasks: 0
Key cache: disabled
Row cache capacity: 10000
Row cache size: 4
Row cache hit rate: 0.2302158273381295
Compacted row minimum size: 87
Compacted row maximum size: 35425
Compacted row mean size: 139
Column Family: HotelMessage
SSTable count: 1
Space used (live): 349735
Space used (total): 349735
Number of Keys (estimate): 512
Memtable Columns Count: 19424
Memtable Data Size: 3408866
Memtable Switch Count: 1
Read Count: 4726
Read Latency: 0.059 ms.
Write Count: 9431
Write Latency: 0.739 ms.
Pending Tasks: 0
Key cache: disabled
Row cache capacity: 10000
Row cache size: 144
Row cache hit rate: 0.968049090139653
Compacted row minimum size: 87
Compacted row maximum size: 24601
Compacted row mean size: 867
Column Family: SavedHotels
SSTable count: 1
Space used (live): 650153
Space used (total): 650153
Number of Keys (estimate): 3456
Memtable Columns Count: 13
Memtable Data Size: 728
Memtable Switch Count: 1
Read Count: 4282
Read Latency: 0.029 ms.
Write Count: 15
Write Latency: 0.014 ms.
Pending Tasks: 0
Key cache: disabled
Row cache capacity: 10000
Row cache size: 13
Row cache hit rate: 0.06819243344231668
Compacted row minimum size: 104
Compacted row maximum size: 2299
Compacted row mean size: 160
Column Family: SavedHotelsInverted
SSTable count: 1
Space used (live): 646988
Space used (total): 646988
Number of Keys (estimate): 3456
Memtable Columns Count: 13
Memtable Data Size: 728
Memtable Switch Count: 1
Read Count: 13
Read Latency: 3.014 ms.
Write Count: 15
Write Latency: 0.007 ms.
Pending Tasks: 0
Key cache: disabled
Row cache capacity: 10000
Row cache size: 2
Row cache hit rate: 0.15384615384615385
Compacted row minimum size: 104
Compacted row maximum size: 2299
Compacted row mean size: 160
Column Family: LastViewedHotelsInverted
SSTable count: 4
Space used (live): 147678370
Space used (total): 147678370
Number of Keys (estimate): 770048
Memtable Columns Count: 2486
Memtable Data Size: 2524930
Memtable Switch Count: 1
Read Count: 2667
Read Latency: 0.622 ms.
Write Count: 3590
Write Latency: 0.010 ms.
Pending Tasks: 0
Key cache: disabled
Row cache capacity: 10000
Row cache size: 477
Row cache hit rate: 0.3820772403449569
Compacted row minimum size: 36
Compacted row maximum size: 51012
Compacted row mean size: 160
Column Family: SavedSearchRequest
SSTable count: 11
Space used (live): 4560932807
Space used (total): 4560932807
Number of Keys (estimate): 1022336
Memtable Columns Count: 7762
Memtable Data Size: 17281321
Memtable Switch Count: 1
Read Count: 20317
Read Latency: 0.242 ms.
Write Count: 11827
Write Latency: 0.022 ms.
Pending Tasks: 0
Key cache: disabled
Row cache capacity: 10000
Row cache size: 619
Row cache hit rate: 0.7583304621745336
Compacted row minimum size: 925
Compacted row maximum size: 1955666
Compacted row mean size: 5014
Column Family: HotelTariffs
SSTable count: 3
Space used (live): 42775204
Space used (total): 42775204
Number of Keys (estimate): 18176
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache: disabled
Row cache capacity: 10000
Row cache size: 0
Row cache hit rate: NaN
Compacted row minimum size: 180
Compacted row maximum size: 9887
Compacted row mean size: 2179
I would appreciate a quick help.
Regards,
Shubham
|