incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Maximum Row Size in Cassandra : Potential Bottleneck
Date Wed, 22 Feb 2012 20:30:49 GMT
I'm not familiar with the solandra data model, but they are wide rows. 

In the past I've done the following to reduce GC pressure with wide rows where I thought GC
was associated with compaction. It's heavy handed and may put pressure on the IO system. Once
things are stabilised you may want to undo some of the changes.

* modify cassandra-env.sh to make sure the new heap is no more than 800M .
* modify cassandra.yaml and change concurrent_compactors to 2 
* modify cassandra.yaml and reduce in_memory_compaction_limit_in_mb to 32 

Then start watching the logs. 

Hope that helps. 

 

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/02/2012, at 10:39 PM, Shubham Srivastava wrote:

> Hi,
>  
> I am using cassandra0.8.6 for saving inverted indexes through Solr(Solr3.3 + Cassandra).
>  
> I am having problem while doing writes which causes  lot of GC activity where CPU usage
increases + heap size usage and my nodes goes on a standstill state dropping most of reads
and writes as per cfstats given below.
>  
> Looking at the row sizes of some of CF’s it seems to be on the higher side right from
average row size of TL from 77 mb with max row size as 186 mb.Typically TL family is used
for writing various terms of an indexed field and so on.
>  
> While doing writes to Cassandra we update/write back this column family + others as well.
Is the row size like of TL normal or is it on the upper side.
>  
> My system can sustain small writes which actually happened after reducing the in in_memory_compaction_limit_in_mb:
to 2 from 64 and read_repair_chance to 0.1 from 1.
>  
> I have a 7 node setup with RF:7 and Read CL:1 with 8 core processor and 14GB RAM. Hepa
size is 8Gb on all the nodes.
>  
>  
>  
>  
> CFSTATS:
>  
> Keyspace: system
>                 Read Count: 19
>                 Read Latency: 8.130157894736843 ms.
>                 Write Count: 31
>                 Write Latency: 0.03029032258064516 ms.
>                 Pending Tasks: 0
>                                 Column Family: NodeIdInfo
>                                 SSTable count: 1
>                                 Space used (live): 4516
>                                 Space used (total): 4516
>                                 Number of Keys (estimate): 128
>                                 Memtable Columns Count: 0
>                                 Memtable Data Size: 0
>                                 Memtable Switch Count: 0
>                                 Read Count: 0
>                                 Read Latency: NaN ms.
>                                 Write Count: 0
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache capacity: 1
>                                 Key cache size: 0
>                                 Key cache hit rate: NaN
>                                 Row cache: disabled
>                                 Compacted row minimum size: 87
>                                 Compacted row maximum size: 103
>                                 Compacted row mean size: 103
>  
>                                 Column Family: HintsColumnFamily
>                                 SSTable count: 2
>                                 Space used (live): 9880
>                                 Space used (total): 34550
>                                 Number of Keys (estimate): 256
>                                 Memtable Columns Count: 0
>                                 Memtable Data Size: 0
>                                 Memtable Switch Count: 5
>                                 Read Count: 10
>                                 Read Latency: NaN ms.
>                                 Write Count: 28
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache capacity: 2
>                                 Key cache size: 2
>                                 Key cache hit rate: NaN
>                                 Row cache: disabled
>                                 Compacted row minimum size: 125
>                                 Compacted row maximum size: 149
>                                 Compacted row mean size: 149
>  
>                                 Column Family: Schema
>                                 SSTable count: 1
>                                 Space used (live): 192414
>                                 Space used (total): 192414
>                                 Number of Keys (estimate): 128
>                                 Memtable Columns Count: 0
>                                 Memtable Data Size: 0
>                                 Memtable Switch Count: 0
>                                 Read Count: 3
>                                 Read Latency: NaN ms.
>                                 Write Count: 0
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache capacity: 1
>                                 Key cache size: 1
>                                 Key cache hit rate: NaN
>                                 Row cache: disabled
>                                 Compacted row minimum size: 104
>                                 Compacted row maximum size: 8239
>                                 Compacted row mean size: 7794
>  
>                                 Column Family: Migrations
>                                 SSTable count: 1
>                                 Space used (live): 300650
>                                 Space used (total): 300650
>                                 Number of Keys (estimate): 128
>                                 Memtable Columns Count: 0
>                                 Memtable Data Size: 0
>                                 Memtable Switch Count: 0
>                                 Read Count: 0
>                                 Read Latency: NaN ms.
>                                 Write Count: 0
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache capacity: 1
>                                 Key cache size: 0
>                                 Key cache hit rate: NaN
>                                 Row cache: disabled
>                                 Compacted row minimum size: 263211
>                                 Compacted row maximum size: 315852
>                                 Compacted row mean size: 315852
>  
>                                 Column Family: IndexInfo
>                                 SSTable count: 0
>                                 Space used (live): 0
>                                 Space used (total): 0
>                                 Number of Keys (estimate): 0
>                                 Memtable Columns Count: 0
>                                 Memtable Data Size: 0
>                                 Memtable Switch Count: 0
>                                 Read Count: 0
>                                 Read Latency: NaN ms.
>                                 Write Count: 0
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache capacity: 1
>                                 Key cache size: 0
>                                 Key cache hit rate: NaN
>                                 Row cache: disabled
>                                 Compacted row minimum size: 0
>                                 Compacted row maximum size: 0
>                                 Compacted row mean size: 0
>  
>                                 Column Family: LocationInfo
>                                 SSTable count: 2
>                                 Space used (live): 11496
>                                 Space used (total): 11496
>                                 Number of Keys (estimate): 256
>                                 Memtable Columns Count: 0
>                                 Memtable Data Size: 0
>                                 Memtable Switch Count: 2
>                                 Read Count: 6
>                                 Read Latency: NaN ms.
>                                 Write Count: 3
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache capacity: 2
>                                 Key cache size: 2
>                                 Key cache hit rate: NaN
>                                 Row cache: disabled
>                                 Compacted row minimum size: 73
>                                 Compacted row maximum size: 310
>                                 Compacted row mean size: 146
>  
> ----------------
> Keyspace: L
>                 Read Count: 343746
>                 Read Latency: 5.11407672525644 ms.
>                 Write Count: 541480
>                 Write Latency: 0.023806109182241266 ms.
>                 Pending Tasks: 0
>                                 Column Family: FC
>                                 SSTable count: 4
>                                 Space used (live): 353742759
>                                 Space used (total): 370460012
>                                 Number of Keys (estimate): 512
>                                 Memtable Columns Count: 64604
>                                 Memtable Data Size: 65810913
>                                 Memtable Switch Count: 4
>                                 Read Count: 93
>                                 Read Latency: NaN ms.
>                                 Write Count: 138640
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 104
>                                 Compacted row maximum size: 10090808
>                                 Compacted row mean size: 2650773
>  
>                                 Column Family: Docs
>                                 SSTable count: 9
>                                 Space used (live): 1645341386
>                                 Space used (total): 1726113393
>                                 Number of Keys (estimate): 356608
>                                 Memtable Columns Count: 65268
>                                 Memtable Data Size: 27459963
>                                 Memtable Switch Count: 4
>                                 Read Count: 74566
>                                 Read Latency: 1.995 ms.
>                                 Write Count: 4688
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 87
>                                 Compacted row maximum size: 51012
>                                 Compacted row mean size: 5557
>  
>                                 Column Family: SI
>                                 SSTable count: 4
>                                 Space used (live): 88904425
>                                 Space used (total): 105431969
>                                 Number of Keys (estimate): 288512
>                                 Memtable Columns Count: 1141
>                                 Memtable Data Size: 767070
>                                 Memtable Switch Count: 4
>                                 Read Count: 8473
>                                 Read Latency: 0.615 ms.
>                                 Write Count: 2447
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 104
>                                 Compacted row maximum size: 25109160
>                                 Compacted row mean size: 4694
>  
>                                 Column Family: TL
>                                 SSTable count: 5
>                                 Space used (live): 334547547
>                                 Space used (total): 348394559
>                                 Number of Keys (estimate): 640
>                                 Memtable Columns Count: 90495
>                                 Memtable Data Size: 8602591
>                                 Memtable Switch Count: 3
>                                 Read Count: 5236
>                                 Read Latency: 30.199 ms.
>                                 Write Count: 2335
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 454827
>                                 Compacted row maximum size: 186563160
>                                 Compacted row mean size: 77287206
>  
>                                 Column Family: TI
>                                 SSTable count: 10
>                                 Space used (live): 3056043044
>                                 Space used (total): 3084009723
>                                 Number of Keys (estimate): 8494720
>                                 Memtable Columns Count: 181003
>                                 Memtable Data Size: 212200530
>                                 Memtable Switch Count: 4
>                                 Read Count: 255378
>                                 Read Latency: 0.962 ms.
>                                 Write Count: 393370
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 104
>                                 Compacted row maximum size: 7007506
>                                 Compacted row mean size: 633
>  
> ----------------
> Keyspace: LH
>                 Read Count: 40729
>                 Read Latency: 1.1628788332637678 ms.
>                 Write Count: 265328
>                 Write Latency: 0.01778465898812036 ms.
>                 Pending Tasks: 0
>                                 Column Family: UserPrefrences
>                                 SSTable count: 6
>                                 Space used (live): 639728096
>                                 Space used (total): 786966376
>                                 Number of Keys (estimate): 2790912
>                                 Memtable Columns Count: 14546
>                                 Memtable Data Size: 7532750
>                                 Memtable Switch Count: 9
>                                 Read Count: 18874
>                                 Read Latency: 0.483 ms.
>                                 Write Count: 107364
>                                 Write Latency: 0.015 ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 87
>                                 Compacted row maximum size: 372
>                                 Compacted row mean size: 239
>  
>                                 Column Family: LastViewedHotels
>                                 SSTable count: 4
>                                 Space used (live): 218367054
>                                 Space used (total): 227203324
>                                 Number of Keys (estimate): 1096192
>                                 Memtable Columns Count: 3724
>                                 Memtable Data Size: 5134228
>                                 Memtable Switch Count: 9
>                                 Read Count: 6085
>                                 Read Latency: 0.939 ms.
>                                 Write Count: 18849
>                                 Write Latency: 0.022 ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 73
>                                 Compacted row maximum size: 51012
>                                 Compacted row mean size: 186
>  
>                                 Column Family: BookedHotels
>                                 SSTable count: 2
>                                 Space used (live): 8288377
>                                 Space used (total): 24867472
>                                 Number of Keys (estimate): 44928
>                                 Memtable Columns Count: 24
>                                 Memtable Data Size: 46264
>                                 Memtable Switch Count: 8
>                                 Read Count: 111
>                                 Read Latency: 0.025 ms.
>                                 Write Count: 209
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 87
>                                 Compacted row maximum size: 35425
>                                 Compacted row mean size: 152
>  
>                                 Column Family: HotelMessage
>                                 SSTable count: 3
>                                 Space used (live): 683567
>                                 Space used (total): 2143936
>                                 Number of Keys (estimate): 1536
>                                 Memtable Columns Count: 14536
>                                 Memtable Data Size: 3764728
>                                 Memtable Switch Count: 8
>                                 Read Count: 4804
>                                 Read Latency: 2.714 ms.
>                                 Write Count: 64967
>                                 Write Latency: 0.012 ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 87
>                                 Compacted row maximum size: 20501
>                                 Compacted row mean size: 496
>  
>                                 Column Family: SavedHotels
>                                 SSTable count: 3
>                                 Space used (live): 1035147
>                                 Space used (total): 3097633
>                                 Number of Keys (estimate): 5760
>                                 Memtable Columns Count: 8
>                                 Memtable Data Size: 15376
>                                 Memtable Switch Count: 7
>                                 Read Count: 3971
>                                 Read Latency: 0.023 ms.
>                                 Write Count: 73
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 104
>                                 Compacted row maximum size: 2299
>                                 Compacted row mean size: 170
>  
>                                 Column Family: SavedHotelsInverted
>                                 SSTable count: 1
>                                 Space used (live): 1022963
>                                 Space used (total): 4108255
>                                 Number of Keys (estimate): 5504
>                                 Memtable Columns Count: 8
>                                 Memtable Data Size: 450
>                                 Memtable Switch Count: 7
>                                 Read Count: 3
>                                 Read Latency: NaN ms.
>                                 Write Count: 73
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 104
>                                 Compacted row maximum size: 2299
>                                 Compacted row mean size: 162
>  
>                                 Column Family: LastViewedHotelsInverted
>                                 SSTable count: 4
>                                 Space used (live): 243688384
>                                 Space used (total): 362107541
>                                 Number of Keys (estimate): 1257216
>                                 Memtable Columns Count: 2917
>                                 Memtable Data Size: 4254486
>                                 Memtable Switch Count: 9
>                                 Read Count: 2454
>                                 Read Latency: 0.091 ms.
>                                 Write Count: 18857
>                                 Write Latency: 0.018 ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 73
>                                 Compacted row maximum size: 51012
>                                 Compacted row mean size: 170
>  
>                                 Column Family: SavedSearchRequest
>                                 SSTable count: 7
>                                 Space used (live): 7125532732
>                                 Space used (total): 7266974627
>                                 Number of Keys (estimate): 1526144
>                                 Memtable Columns Count: 8854
>                                 Memtable Data Size: 23875680
>                                 Memtable Switch Count: 9
>                                 Read Count: 4427
>                                 Read Latency: 5.022 ms.
>                                 Write Count: 54938
>                                 Write Latency: 0.026 ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 925
>                                 Compacted row maximum size: 4055269
>                                 Compacted row mean size: 4912
>  
>                                 Column Family: HotelTariffs
>                                 SSTable count: 1
>                                 Space used (live): 46306159
>                                 Space used (total): 46306159
>                                 Number of Keys (estimate): 7680
>                                 Memtable Columns Count: 0
>                                 Memtable Data Size: 0
>                                 Memtable Switch Count: 0
>                                 Read Count: 0
>                                 Read Latency: NaN ms.
>                                 Write Count: 0
>                                 Write Latency: NaN ms.
>                                 Pending Tasks: 0
>                                 Key cache: disabled
>                                 Row cache: disabled
>                                 Compacted row minimum size: 180
>                                 Compacted row maximum size: 9887
>                                 Compacted row mean size: 6535
>  
>  
>  
>  
> Regards,
> Shubham Srivastava | Technical Lead - Technology Development
> +91 124 439 5428   |  MakeMyTrip.com, 103 Udyog Vihar Phase 1, Gurgaon, Haryana - 122
016, India
> <image002.png>What's new? My Trip Rewards - An exclusive loyalty program for MakeMyTrip
customers.
> <image003.gif>
> <image004.gif>
> Office Map
> <image005.gif>
> Facebook
> <image006.gif>
> Twitter
>  


Mime
View raw message