cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Srivastava <Shubham.Srivast...@makemytrip.com>
Subject Cassandra Compaction is Killing
Date Sat, 04 Feb 2012 04:58:48 GMT
I have a cassandra setup with 7 Node Ring single DC with RF:7 and Read:CL:1.There is live traffic
on these nodes except one -> the traffic is 90% read. There are also writes happening to
all these nodes which typically are user specific data etc.These nodes aNow at times what
is happening is there are 2-3 say nodes getting in a hanged state and eventually the whole
ring behaves so. There state in terms of CPU Usage is very High 17Loads ,network incoming
and outgoing requests packet were very high and there is continuous GC(Major and Minor) pauses
happening and Read Messages dropped  simultaneously.Also the SSTables couunt have increased
and decreased for some families during this time. So majorly its the GC and SStable compaction
and Memtable flush happening. We are running cassandra behind solr using Solandra.The max
docs that we have are around 0.5Million.

We keep one node as the main write node where we run sheduled job thats pools data into the
cluster from MySql db.These jobs are run hourly.Recently we have increased the data to somewhere
to the size of 30X . Before the same setup was stable and these jobs used to run in every
5 minutes apart from the similar problem 2-3 times.

GC:Settings
4Gb Heap Xmx,Xms
2Gb Young Geneartion: Xmn
ParNew
CMS
 -XX:+UseParNewGC" 
 -XX:+UseConcMarkSweepGC" 
 -XX:+CMSParallelRemarkEnabled" 
 -XX:SurvivorRatio=8" 
 -XX:MaxTenuringThreshold=1"
 -XX:CMSInitiatingOccupancyFraction=75"
 -XX:+UseCMSInitiatingOccupancyOnly"
 
 Memtable Threshold is also:0.75 and rest default cassandra settings
 
 Also we dont have in place a connectionpool for thrift.We are using cassandra:0.8.6 with
solr:3.3
 
 Will share the Cfstats shortly or anything else needed for that matter.Can you guys help
me kn this.
 
 
 ==========================================
 
 
 Keyspace: system
         Read Count: 19
         Read Latency: 5.6876842105263155 ms.
         Write Count: 2375
         Write Latency: 0.010477894736842104 ms.
         Pending Tasks: 0
                 Column Family: NodeIdInfo
                 SSTable count: 0
                 Space used (live): 0
                 Space used (total): 0
                 Number of Keys (estimate): 0
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 0
                 Read Count: 0
                 Read Latency: NaN ms.
                 Write Count: 0
                 Write Latency: NaN ms.
                 Pending Tasks: 0
                 Key cache capacity: 1
                 Key cache size: 0
                 Key cache hit rate: NaN
                 Row cache: disabled
                 Compacted row minimum size: 0
                 Compacted row maximum size: 0
                 Compacted row mean size: 0
 
                 Column Family: HintsColumnFamily
                 SSTable count: 2
                 Space used (live): 492064
                 Space used (total): 492064
                 Number of Keys (estimate): 256
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 3
                 Read Count: 10
                 Read Latency: 5.643 ms.
                 Write Count: 2372
                 Write Latency: 0.010 ms.
                 Pending Tasks: 0
                 Key cache capacity: 2
                 Key cache size: 2
                 Key cache hit rate: 0.42857142857142855
                 Row cache: disabled
                 Compacted row minimum size: 219343
                 Compacted row maximum size: 263210
                 Compacted row mean size: 263210
 
                 Column Family: Schema
                 SSTable count: 2
                 Space used (live): 20827
                 Space used (total): 20827
                 Number of Keys (estimate): 256
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 0
                 Read Count: 3
                 Read Latency: 4.253 ms.
                 Write Count: 0
                 Write Latency: NaN ms.
                 Pending Tasks: 0
                 Key cache capacity: 2
                 Key cache size: 2
                 Key cache hit rate: 0.0
                 Row cache: disabled
                 Compacted row minimum size: 104
                 Compacted row maximum size: 8239
                 Compacted row mean size: 3314
 
                 Column Family: Migrations
                 SSTable count: 2
                 Space used (live): 33180
                 Space used (total): 33180
                 Number of Keys (estimate): 256
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 0
                 Read Count: 0
                 Read Latency: NaN ms.
                 Write Count: 0
                 Write Latency: NaN ms.
                 Pending Tasks: 0
                 Key cache capacity: 2
                 Key cache size: 0
                 Key cache hit rate: NaN
                 Row cache: disabled
                 Compacted row minimum size: 9888
                 Compacted row maximum size: 17084
                 Compacted row mean size: 14474
 
                 Column Family: IndexInfo
                 SSTable count: 0
                 Space used (live): 0
                 Space used (total): 0
                 Number of Keys (estimate): 0
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 0
                 Read Count: 0
                 Read Latency: NaN ms.
                 Write Count: 0
                 Write Latency: NaN ms.
                 Pending Tasks: 0
                 Key cache capacity: 1
                 Key cache size: 0
                 Key cache hit rate: NaN
                 Row cache: disabled
                 Compacted row minimum size: 0
                 Compacted row maximum size: 0
                 Compacted row mean size: 0
 
                 Column Family: LocationInfo
                 SSTable count: 3
                 Space used (live): 15844
                 Space used (total): 15844
                 Number of Keys (estimate): 384
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 2
                 Read Count: 6
                 Read Latency: 6.479 ms.
                 Write Count: 3
                 Write Latency: 0.004 ms.
                 Pending Tasks: 0
                 Key cache capacity: 3
                 Key cache size: 3
                 Key cache hit rate: 0.3333333333333333
                 Row cache: disabled
                 Compacted row minimum size: 73
                 Compacted row maximum size: 310
                 Compacted row mean size: 126
 
 ----------------
 Keyspace: L
         Read Count: 535675
         Read Latency: 4.472875136976712 ms.
         Write Count: 282216
         Write Latency: 0.08156946806701251 ms.
         Pending Tasks: 0
                 Column Family: FC
                 SSTable count: 6
                 Space used (live): 444790202
                 Space used (total): 444790202
                 Number of Keys (estimate): 768
                 Memtable Columns Count: 93231
                 Memtable Data Size: 172790560
                 Memtable Switch Count: 1
                 Read Count: 6651
                 Read Latency: 62.041 ms.
                 Write Count: 94235
                 Write Latency: 0.026 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 104
                 Compacted row maximum size: 10090808
                 Compacted row mean size: 4256880
 
                 Column Family: Docs
                 SSTable count: 7
                 Space used (live): 1487268134
                 Space used (total): 1487268134
                 Number of Keys (estimate): 303744
                 Memtable Columns Count: 98040
                 Memtable Data Size: 37732086
                 Memtable Switch Count: 1
                 Read Count: 125986
                 Read Latency: 1.868 ms.
                 Write Count: 11900
                 Write Latency: 0.520 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 36
                 Compacted row maximum size: 51012
                 Compacted row mean size: 5547
 
                 Column Family: SI
                 SSTable count: 3
                 Space used (live): 136512555
                 Space used (total): 136512555
                 Number of Keys (estimate): 286848
                 Memtable Columns Count: 5344
                 Memtable Data Size: 21139811
                 Memtable Switch Count: 1
                 Read Count: 27642
                 Read Latency: 0.880 ms.
                 Write Count: 5815
                 Write Latency: 0.093 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 73
                 Compacted row maximum size: 30130992
                 Compacted row mean size: 5022190
 
                 Column Family: TL
                 SSTable count: 4
                 Space used (live): 314821524
                 Space used (total): 314821524
                 Number of Keys (estimate): 512
                 Memtable Columns Count: 83428
                 Memtable Data Size: 16069959
                 Memtable Switch Count: 1
                 Read Count: 10867
                 Read Latency: 17.949 ms.
                 Write Count: 5386
                 Write Latency: 0.578 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 12108971
                 Compacted row maximum size: 186563160
                 Compacted row mean size: 90195666
 
                 Column Family: TI
                 SSTable count: 6
                 Space used (live): 1451685937
                 Space used (total): 1451685937
                 Number of Keys (estimate): 3910144
                 Memtable Columns Count: 166006
                 Memtable Data Size: 184308808
                 Memtable Switch Count: 1
                 Read Count: 364529
                 Read Latency: 4.194 ms.
                 Write Count: 164880
                 Write Latency: 0.065 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 104
                 Compacted row maximum size: 7007506
                 Compacted row mean size: 18922
 
 ----------------
 Keyspace: LH
         Read Count: 61353
         Read Latency: 0.20730264208759147 ms.
         Write Count: 56289
         Write Latency: 0.13461228659240704 ms.
         Pending Tasks: 0
                 Column Family: UserPrefrences
                 SSTable count: 4
                 Space used (live): 335309522
                 Space used (total): 335309522
                 Number of Keys (estimate): 1513600
                 Memtable Columns Count: 12807
                 Memtable Data Size: 9144714
                 Memtable Switch Count: 1
                 Read Count: 20132
                 Read Latency: 0.201 ms.
                 Write Count: 27817
                 Write Latency: 0.009 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 61
                 Compacted row maximum size: 215
                 Compacted row mean size: 179
 
                 Column Family: LastViewedHotels
                 SSTable count: 3
                 Space used (live): 152316912
                 Space used (total): 152316912
                 Number of Keys (estimate): 767104
                 Memtable Columns Count: 3059
                 Memtable Data Size: 2906758
                 Memtable Switch Count: 1
                 Read Count: 9077
                 Read Latency: 0.180 ms.
                 Write Count: 3585
                 Write Latency: 0.018 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 565
                 Row cache hit rate: 0.7896882229811611
                 Compacted row minimum size: 36
                 Compacted row maximum size: 51012
                 Compacted row mean size: 144
 
                 Column Family: BookedHotels
                 SSTable count: 3
                 Space used (live): 7274700
                 Space used (total): 7274700
                 Number of Keys (estimate): 39680
                 Memtable Columns Count: 7
                 Memtable Data Size: 392
                 Memtable Switch Count: 1
                 Read Count: 139
                 Read Latency: 0.027 ms.
                 Write Count: 10
                 Write Latency: 0.008 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 4
                 Row cache hit rate: 0.2302158273381295
                 Compacted row minimum size: 87
                 Compacted row maximum size: 35425
                 Compacted row mean size: 139
 
                 Column Family: HotelMessage
                 SSTable count: 1
                 Space used (live): 349735
                 Space used (total): 349735
                 Number of Keys (estimate): 512
                 Memtable Columns Count: 19424
                 Memtable Data Size: 3408866
                 Memtable Switch Count: 1
                 Read Count: 4726
                 Read Latency: 0.059 ms.
                 Write Count: 9431
                 Write Latency: 0.739 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 144
                 Row cache hit rate: 0.968049090139653
                 Compacted row minimum size: 87
                 Compacted row maximum size: 24601
                 Compacted row mean size: 867
 
                 Column Family: SavedHotels
                 SSTable count: 1
                 Space used (live): 650153
                 Space used (total): 650153
                 Number of Keys (estimate): 3456
                 Memtable Columns Count: 13
                 Memtable Data Size: 728
                 Memtable Switch Count: 1
                 Read Count: 4282
                 Read Latency: 0.029 ms.
                 Write Count: 15
                 Write Latency: 0.014 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 13
                 Row cache hit rate: 0.06819243344231668
                 Compacted row minimum size: 104
                 Compacted row maximum size: 2299
                 Compacted row mean size: 160
 
                 Column Family: SavedHotelsInverted
                 SSTable count: 1
                 Space used (live): 646988
                 Space used (total): 646988
                 Number of Keys (estimate): 3456
                 Memtable Columns Count: 13
                 Memtable Data Size: 728
                 Memtable Switch Count: 1
                 Read Count: 13
                 Read Latency: 3.014 ms.
                 Write Count: 15
                 Write Latency: 0.007 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 2
                 Row cache hit rate: 0.15384615384615385
                 Compacted row minimum size: 104
                 Compacted row maximum size: 2299
                 Compacted row mean size: 160
 
                 Column Family: LastViewedHotelsInverted
                 SSTable count: 4
                 Space used (live): 147678370
                 Space used (total): 147678370
                 Number of Keys (estimate): 770048
                 Memtable Columns Count: 2486
                 Memtable Data Size: 2524930
                 Memtable Switch Count: 1
                 Read Count: 2667
                 Read Latency: 0.622 ms.
                 Write Count: 3590
                 Write Latency: 0.010 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 477
                 Row cache hit rate: 0.3820772403449569
                 Compacted row minimum size: 36
                 Compacted row maximum size: 51012
                 Compacted row mean size: 160
 
                 Column Family: SavedSearchRequest
                 SSTable count: 11
                 Space used (live): 4560932807
                 Space used (total): 4560932807
                 Number of Keys (estimate): 1022336
                 Memtable Columns Count: 7762
                 Memtable Data Size: 17281321
                 Memtable Switch Count: 1
                 Read Count: 20317
                 Read Latency: 0.242 ms.
                 Write Count: 11827
                 Write Latency: 0.022 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 619
                 Row cache hit rate: 0.7583304621745336
                 Compacted row minimum size: 925
                 Compacted row maximum size: 1955666
                 Compacted row mean size: 5014
 
                 Column Family: HotelTariffs
                 SSTable count: 3
                 Space used (live): 42775204
                 Space used (total): 42775204
                 Number of Keys (estimate): 18176
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 0
                 Read Count: 0
                 Read Latency: NaN ms.
                 Write Count: 0
                 Write Latency: NaN ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 0
                 Row cache hit rate: NaN
                 Compacted row minimum size: 180
                 Compacted row maximum size: 9887
                 Compacted row mean size: 2179

I would appreciate a quick help.

Regards,
Shubham

Mime
View raw message