incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra Compaction is Killing
Date Mon, 06 Feb 2012 19:21:12 GMT
Sounds like GC is the first problem to tackle. Can you give the process more memory ?  

Look at the logs and work out what the GC is doing, the cassandra logs are not exactly the
same as this but you get the idea http://blogs.oracle.com/poonam/entry/understanding_cms_gc_logs

The next thing I normally do is revert all changes to the default memory and GC settings.


When you say memtable threshold is 0.75 do you mean the memtable_total_space_in_mb setting
in yaml ? 
Is Cassandra logging messages about the the heap been full, what are they saying ? 

Wide rows, such as the TL column family in the L keyspace can increase the about of GC work
that goes on. This can be somewhat alleviated by reducing the in_memory_compaction_limit_in_mb
, this will increase the amount of IO that is needed. 

Heavy write traffic can also increase GC activity. Watch the tp stats during the load process,
are you overloading the cluster ? Do you see pending tasks backing up ? 
Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 4/02/2012, at 5:58 PM, Shubham Srivastava wrote:

> I have a cassandra setup with 7 Node Ring single DC with RF:7 and Read:CL:1.There is
live traffic on these nodes except one -> the traffic is 90% read. There are also writes
happening to all these nodes which typically are user specific data etc.These nodes aNow at
times what is happening is there are 2-3 say nodes getting in a hanged state and eventually
the whole ring behaves so. There state in terms of CPU Usage is very High 17Loads ,network
incoming and outgoing requests packet were very high and there is continuous GC(Major and
Minor) pauses happening and Read Messages dropped  simultaneously.Also the SSTables couunt
have increased and decreased for some families during this time. So majorly its the GC and
SStable compaction and Memtable flush happening. We are running cassandra behind solr using
Solandra.The max docs that we have are around 0.5Million.
> 
> We keep one node as the main write node where we run sheduled job thats pools data into
the cluster from MySql db.These jobs are run hourly.Recently we have increased the data to
somewhere to the size of 30X . Before the same setup was stable and these jobs used to run
in every 5 minutes apart from the similar problem 2-3 times.
> 
> GC:Settings
> 4Gb Heap Xmx,Xms
> 2Gb Young Geneartion: Xmn
> ParNew
> CMS
> -XX:+UseParNewGC" 
> -XX:+UseConcMarkSweepGC" 
> -XX:+CMSParallelRemarkEnabled" 
> -XX:SurvivorRatio=8" 
> -XX:MaxTenuringThreshold=1"
> -XX:CMSInitiatingOccupancyFraction=75"
> -XX:+UseCMSInitiatingOccupancyOnly"
> 
> Memtable Threshold is also:0.75 and rest default cassandra settings
> 
> Also we dont have in place a connectionpool for thrift.We are using cassandra:0.8.6 with
solr:3.3
> 
> Will share the Cfstats shortly or anything else needed for that matter.Can you guys help
me kn this.
> 
> 
> ==========================================
> 
> 
> Keyspace: system
>         Read Count: 19
>         Read Latency: 5.6876842105263155 ms.
>         Write Count: 2375
>         Write Latency: 0.010477894736842104 ms.
>         Pending Tasks: 0
>                 Column Family: NodeIdInfo
>                 SSTable count: 0
>                 Space used (live): 0
>                 Space used (total): 0
>                 Number of Keys (estimate): 0
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 0
>                 Read Count: 0
>                 Read Latency: NaN ms.
>                 Write Count: 0
>                 Write Latency: NaN ms.
>                 Pending Tasks: 0
>                 Key cache capacity: 1
>                 Key cache size: 0
>                 Key cache hit rate: NaN
>                 Row cache: disabled
>                 Compacted row minimum size: 0
>                 Compacted row maximum size: 0
>                 Compacted row mean size: 0
> 
>                 Column Family: HintsColumnFamily
>                 SSTable count: 2
>                 Space used (live): 492064
>                 Space used (total): 492064
>                 Number of Keys (estimate): 256
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 3
>                 Read Count: 10
>                 Read Latency: 5.643 ms.
>                 Write Count: 2372
>                 Write Latency: 0.010 ms.
>                 Pending Tasks: 0
>                 Key cache capacity: 2
>                 Key cache size: 2
>                 Key cache hit rate: 0.42857142857142855
>                 Row cache: disabled
>                 Compacted row minimum size: 219343
>                 Compacted row maximum size: 263210
>                 Compacted row mean size: 263210
> 
>                 Column Family: Schema
>                 SSTable count: 2
>                 Space used (live): 20827
>                 Space used (total): 20827
>                 Number of Keys (estimate): 256
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 0
>                 Read Count: 3
>                 Read Latency: 4.253 ms.
>                 Write Count: 0
>                 Write Latency: NaN ms.
>                 Pending Tasks: 0
>                 Key cache capacity: 2
>                 Key cache size: 2
>                 Key cache hit rate: 0.0
>                 Row cache: disabled
>                 Compacted row minimum size: 104
>                 Compacted row maximum size: 8239
>                 Compacted row mean size: 3314
> 
>                 Column Family: Migrations
>                 SSTable count: 2
>                 Space used (live): 33180
>                 Space used (total): 33180
>                 Number of Keys (estimate): 256
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 0
>                 Read Count: 0
>                 Read Latency: NaN ms.
>                 Write Count: 0
>                 Write Latency: NaN ms.
>                 Pending Tasks: 0
>                 Key cache capacity: 2
>                 Key cache size: 0
>                 Key cache hit rate: NaN
>                 Row cache: disabled
>                 Compacted row minimum size: 9888
>                 Compacted row maximum size: 17084
>                 Compacted row mean size: 14474
> 
>                 Column Family: IndexInfo
>                 SSTable count: 0
>                 Space used (live): 0
>                 Space used (total): 0
>                 Number of Keys (estimate): 0
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 0
>                 Read Count: 0
>                 Read Latency: NaN ms.
>                 Write Count: 0
>                 Write Latency: NaN ms.
>                 Pending Tasks: 0
>                 Key cache capacity: 1
>                 Key cache size: 0
>                 Key cache hit rate: NaN
>                 Row cache: disabled
>                 Compacted row minimum size: 0
>                 Compacted row maximum size: 0
>                 Compacted row mean size: 0
> 
>                 Column Family: LocationInfo
>                 SSTable count: 3
>                 Space used (live): 15844
>                 Space used (total): 15844
>                 Number of Keys (estimate): 384
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 2
>                 Read Count: 6
>                 Read Latency: 6.479 ms.
>                 Write Count: 3
>                 Write Latency: 0.004 ms.
>                 Pending Tasks: 0
>                 Key cache capacity: 3
>                 Key cache size: 3
>                 Key cache hit rate: 0.3333333333333333
>                 Row cache: disabled
>                 Compacted row minimum size: 73
>                 Compacted row maximum size: 310
>                 Compacted row mean size: 126
> 
> ----------------
> Keyspace: L
>         Read Count: 535675
>         Read Latency: 4.472875136976712 ms.
>         Write Count: 282216
>         Write Latency: 0.08156946806701251 ms.
>         Pending Tasks: 0
>                 Column Family: FC
>                 SSTable count: 6
>                 Space used (live): 444790202
>                 Space used (total): 444790202
>                 Number of Keys (estimate): 768
>                 Memtable Columns Count: 93231
>                 Memtable Data Size: 172790560
>                 Memtable Switch Count: 1
>                 Read Count: 6651
>                 Read Latency: 62.041 ms.
>                 Write Count: 94235
>                 Write Latency: 0.026 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache: disabled
>                 Compacted row minimum size: 104
>                 Compacted row maximum size: 10090808
>                 Compacted row mean size: 4256880
> 
>                 Column Family: Docs
>                 SSTable count: 7
>                 Space used (live): 1487268134
>                 Space used (total): 1487268134
>                 Number of Keys (estimate): 303744
>                 Memtable Columns Count: 98040
>                 Memtable Data Size: 37732086
>                 Memtable Switch Count: 1
>                 Read Count: 125986
>                 Read Latency: 1.868 ms.
>                 Write Count: 11900
>                 Write Latency: 0.520 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache: disabled
>                 Compacted row minimum size: 36
>                 Compacted row maximum size: 51012
>                 Compacted row mean size: 5547
> 
>                 Column Family: SI
>                 SSTable count: 3
>                 Space used (live): 136512555
>                 Space used (total): 136512555
>                 Number of Keys (estimate): 286848
>                 Memtable Columns Count: 5344
>                 Memtable Data Size: 21139811
>                 Memtable Switch Count: 1
>                 Read Count: 27642
>                 Read Latency: 0.880 ms.
>                 Write Count: 5815
>                 Write Latency: 0.093 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache: disabled
>                 Compacted row minimum size: 73
>                 Compacted row maximum size: 30130992
>                 Compacted row mean size: 5022190
> 
>                 Column Family: TL
>                 SSTable count: 4
>                 Space used (live): 314821524
>                 Space used (total): 314821524
>                 Number of Keys (estimate): 512
>                 Memtable Columns Count: 83428
>                 Memtable Data Size: 16069959
>                 Memtable Switch Count: 1
>                 Read Count: 10867
>                 Read Latency: 17.949 ms.
>                 Write Count: 5386
>                 Write Latency: 0.578 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache: disabled
>                 Compacted row minimum size: 12108971
>                 Compacted row maximum size: 186563160
>                 Compacted row mean size: 90195666
> 
>                 Column Family: TI
>                 SSTable count: 6
>                 Space used (live): 1451685937
>                 Space used (total): 1451685937
>                 Number of Keys (estimate): 3910144
>                 Memtable Columns Count: 166006
>                 Memtable Data Size: 184308808
>                 Memtable Switch Count: 1
>                 Read Count: 364529
>                 Read Latency: 4.194 ms.
>                 Write Count: 164880
>                 Write Latency: 0.065 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache: disabled
>                 Compacted row minimum size: 104
>                 Compacted row maximum size: 7007506
>                 Compacted row mean size: 18922
> 
> ----------------
> Keyspace: LH
>         Read Count: 61353
>         Read Latency: 0.20730264208759147 ms.
>         Write Count: 56289
>         Write Latency: 0.13461228659240704 ms.
>         Pending Tasks: 0
>                 Column Family: UserPrefrences
>                 SSTable count: 4
>                 Space used (live): 335309522
>                 Space used (total): 335309522
>                 Number of Keys (estimate): 1513600
>                 Memtable Columns Count: 12807
>                 Memtable Data Size: 9144714
>                 Memtable Switch Count: 1
>                 Read Count: 20132
>                 Read Latency: 0.201 ms.
>                 Write Count: 27817
>                 Write Latency: 0.009 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache: disabled
>                 Compacted row minimum size: 61
>                 Compacted row maximum size: 215
>                 Compacted row mean size: 179
> 
>                 Column Family: LastViewedHotels
>                 SSTable count: 3
>                 Space used (live): 152316912
>                 Space used (total): 152316912
>                 Number of Keys (estimate): 767104
>                 Memtable Columns Count: 3059
>                 Memtable Data Size: 2906758
>                 Memtable Switch Count: 1
>                 Read Count: 9077
>                 Read Latency: 0.180 ms.
>                 Write Count: 3585
>                 Write Latency: 0.018 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache capacity: 10000
>                 Row cache size: 565
>                 Row cache hit rate: 0.7896882229811611
>                 Compacted row minimum size: 36
>                 Compacted row maximum size: 51012
>                 Compacted row mean size: 144
> 
>                 Column Family: BookedHotels
>                 SSTable count: 3
>                 Space used (live): 7274700
>                 Space used (total): 7274700
>                 Number of Keys (estimate): 39680
>                 Memtable Columns Count: 7
>                 Memtable Data Size: 392
>                 Memtable Switch Count: 1
>                 Read Count: 139
>                 Read Latency: 0.027 ms.
>                 Write Count: 10
>                 Write Latency: 0.008 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache capacity: 10000
>                 Row cache size: 4
>                 Row cache hit rate: 0.2302158273381295
>                 Compacted row minimum size: 87
>                 Compacted row maximum size: 35425
>                 Compacted row mean size: 139
> 
>                 Column Family: HotelMessage
>                 SSTable count: 1
>                 Space used (live): 349735
>                 Space used (total): 349735
>                 Number of Keys (estimate): 512
>                 Memtable Columns Count: 19424
>                 Memtable Data Size: 3408866
>                 Memtable Switch Count: 1
>                 Read Count: 4726
>                 Read Latency: 0.059 ms.
>                 Write Count: 9431
>                 Write Latency: 0.739 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache capacity: 10000
>                 Row cache size: 144
>                 Row cache hit rate: 0.968049090139653
>                 Compacted row minimum size: 87
>                 Compacted row maximum size: 24601
>                 Compacted row mean size: 867
> 
>                 Column Family: SavedHotels
>                 SSTable count: 1
>                 Space used (live): 650153
>                 Space used (total): 650153
>                 Number of Keys (estimate): 3456
>                 Memtable Columns Count: 13
>                 Memtable Data Size: 728
>                 Memtable Switch Count: 1
>                 Read Count: 4282
>                 Read Latency: 0.029 ms.
>                 Write Count: 15
>                 Write Latency: 0.014 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache capacity: 10000
>                 Row cache size: 13
>                 Row cache hit rate: 0.06819243344231668
>                 Compacted row minimum size: 104
>                 Compacted row maximum size: 2299
>                 Compacted row mean size: 160
> 
>                 Column Family: SavedHotelsInverted
>                 SSTable count: 1
>                 Space used (live): 646988
>                 Space used (total): 646988
>                 Number of Keys (estimate): 3456
>                 Memtable Columns Count: 13
>                 Memtable Data Size: 728
>                 Memtable Switch Count: 1
>                 Read Count: 13
>                 Read Latency: 3.014 ms.
>                 Write Count: 15
>                 Write Latency: 0.007 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache capacity: 10000
>                 Row cache size: 2
>                 Row cache hit rate: 0.15384615384615385
>                 Compacted row minimum size: 104
>                 Compacted row maximum size: 2299
>                 Compacted row mean size: 160
> 
>                 Column Family: LastViewedHotelsInverted
>                 SSTable count: 4
>                 Space used (live): 147678370
>                 Space used (total): 147678370
>                 Number of Keys (estimate): 770048
>                 Memtable Columns Count: 2486
>                 Memtable Data Size: 2524930
>                 Memtable Switch Count: 1
>                 Read Count: 2667
>                 Read Latency: 0.622 ms.
>                 Write Count: 3590
>                 Write Latency: 0.010 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache capacity: 10000
>                 Row cache size: 477
>                 Row cache hit rate: 0.3820772403449569
>                 Compacted row minimum size: 36
>                 Compacted row maximum size: 51012
>                 Compacted row mean size: 160
> 
>                 Column Family: SavedSearchRequest
>                 SSTable count: 11
>                 Space used (live): 4560932807
>                 Space used (total): 4560932807
>                 Number of Keys (estimate): 1022336
>                 Memtable Columns Count: 7762
>                 Memtable Data Size: 17281321
>                 Memtable Switch Count: 1
>                 Read Count: 20317
>                 Read Latency: 0.242 ms.
>                 Write Count: 11827
>                 Write Latency: 0.022 ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache capacity: 10000
>                 Row cache size: 619
>                 Row cache hit rate: 0.7583304621745336
>                 Compacted row minimum size: 925
>                 Compacted row maximum size: 1955666
>                 Compacted row mean size: 5014
> 
>                 Column Family: HotelTariffs
>                 SSTable count: 3
>                 Space used (live): 42775204
>                 Space used (total): 42775204
>                 Number of Keys (estimate): 18176
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 0
>                 Read Count: 0
>                 Read Latency: NaN ms.
>                 Write Count: 0
>                 Write Latency: NaN ms.
>                 Pending Tasks: 0
>                 Key cache: disabled
>                 Row cache capacity: 10000
>                 Row cache size: 0
>                 Row cache hit rate: NaN
>                 Compacted row minimum size: 180
>                 Compacted row maximum size: 9887
>                 Compacted row mean size: 2179
> 
> I would appreciate a quick help.
> 
> Regards,
> Shubham


Mime
View raw message