cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Srivastava <Shubham.Srivast...@makemytrip.com>
Subject RE: Maximum Row Size in Cassandra : Potential Bottleneck
Date Wed, 22 Feb 2012 11:36:14 GMT
Any suggestions on the same.

From: Shubham Srivastava [mailto:Shubham.Srivastava@makemytrip.com]
Sent: 21 February 2012 15:10
To: user@cassandra.apache.org
Subject: Maximum Row Size in Cassandra : Potential Bottleneck

Hi,

I am using cassandra0.8.6 for saving inverted indexes through Solr(Solr3.3 + Cassandra).

I am having problem while doing writes which causes  lot of GC activity where CPU usage increases
+ heap size usage and my nodes goes on a standstill state dropping most of reads and writes
as per cfstats given below.

Looking at the row sizes of some of CF's it seems to be on the higher side right from average
row size of TL from 77 mb with max row size as 186 mb.Typically TL family is used for writing
various terms of an indexed field and so on.

While doing writes to Cassandra we update/write back this column family + others as well.
Is the row size like of TL normal or is it on the upper side.

My system can sustain small writes which actually happened after reducing the in in_memory_compaction_limit_in_mb:
to 2 from 64 and read_repair_chance to 0.1 from 1.

I have a 7 node setup with RF:7 and Read CL:1 with 8 core processor and 14GB RAM. Hepa size
is 8Gb on all the nodes.




CFSTATS:

Keyspace: system
                Read Count: 19
                Read Latency: 8.130157894736843 ms.
                Write Count: 31
                Write Latency: 0.03029032258064516 ms.
                Pending Tasks: 0
                                Column Family: NodeIdInfo
                                SSTable count: 1
                                Space used (live): 4516
                                Space used (total): 4516
                                Number of Keys (estimate): 128
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 0
                                Read Count: 0
                                Read Latency: NaN ms.
                                Write Count: 0
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 1
                                Key cache size: 0
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 87
                                Compacted row maximum size: 103
                                Compacted row mean size: 103

                                Column Family: HintsColumnFamily
                                SSTable count: 2
                                Space used (live): 9880
                                Space used (total): 34550
                                Number of Keys (estimate): 256
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 5
                                Read Count: 10
                                Read Latency: NaN ms.
                                Write Count: 28
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 2
                                Key cache size: 2
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 125
                                Compacted row maximum size: 149
                                Compacted row mean size: 149

                                Column Family: Schema
                                SSTable count: 1
                                Space used (live): 192414
                                Space used (total): 192414
                                Number of Keys (estimate): 128
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 0
                                Read Count: 3
                                Read Latency: NaN ms.
                                Write Count: 0
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 1
                                Key cache size: 1
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 8239
                                Compacted row mean size: 7794

                                Column Family: Migrations
                                SSTable count: 1
                                Space used (live): 300650
                                Space used (total): 300650
                                Number of Keys (estimate): 128
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 0
                                Read Count: 0
                                Read Latency: NaN ms.
                                Write Count: 0
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 1
                                Key cache size: 0
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 263211
                                Compacted row maximum size: 315852
                                Compacted row mean size: 315852

                                Column Family: IndexInfo
                                SSTable count: 0
                                Space used (live): 0
                                Space used (total): 0
                                Number of Keys (estimate): 0
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 0
                                Read Count: 0
                                Read Latency: NaN ms.
                                Write Count: 0
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 1
                                Key cache size: 0
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 0
                                Compacted row maximum size: 0
                                Compacted row mean size: 0

                                Column Family: LocationInfo
                                SSTable count: 2
                                Space used (live): 11496
                                Space used (total): 11496
                                Number of Keys (estimate): 256
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 2
                                Read Count: 6
                                Read Latency: NaN ms.
                                Write Count: 3
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 2
                                Key cache size: 2
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 73
                                Compacted row maximum size: 310
                                Compacted row mean size: 146

----------------
Keyspace: L
                Read Count: 343746
                Read Latency: 5.11407672525644 ms.
                Write Count: 541480
                Write Latency: 0.023806109182241266 ms.
                Pending Tasks: 0
                                Column Family: FC
                                SSTable count: 4
                                Space used (live): 353742759
                                Space used (total): 370460012
                                Number of Keys (estimate): 512
                                Memtable Columns Count: 64604
                                Memtable Data Size: 65810913
                                Memtable Switch Count: 4
                                Read Count: 93
                                Read Latency: NaN ms.
                                Write Count: 138640
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 10090808
                                Compacted row mean size: 2650773

                                Column Family: Docs
                                SSTable count: 9
                                Space used (live): 1645341386
                                Space used (total): 1726113393
                                Number of Keys (estimate): 356608
                                Memtable Columns Count: 65268
                                Memtable Data Size: 27459963
                                Memtable Switch Count: 4
                                Read Count: 74566
                                Read Latency: 1.995 ms.
                                Write Count: 4688
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 87
                                Compacted row maximum size: 51012
                                Compacted row mean size: 5557

                                Column Family: SI
                                SSTable count: 4
                                Space used (live): 88904425
                                Space used (total): 105431969
                                Number of Keys (estimate): 288512
                                Memtable Columns Count: 1141
                                Memtable Data Size: 767070
                                Memtable Switch Count: 4
                                Read Count: 8473
                                Read Latency: 0.615 ms.
                                Write Count: 2447
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 25109160
                                Compacted row mean size: 4694

                                Column Family: TL
                                SSTable count: 5
                                Space used (live): 334547547
                                Space used (total): 348394559
                                Number of Keys (estimate): 640
                                Memtable Columns Count: 90495
                                Memtable Data Size: 8602591
                                Memtable Switch Count: 3
                                Read Count: 5236
                                Read Latency: 30.199 ms.
                                Write Count: 2335
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 454827
                                Compacted row maximum size: 186563160
                                Compacted row mean size: 77287206

                                Column Family: TI
                                SSTable count: 10
                                Space used (live): 3056043044
                                Space used (total): 3084009723
                                Number of Keys (estimate): 8494720
                                Memtable Columns Count: 181003
                                Memtable Data Size: 212200530
                                Memtable Switch Count: 4
                                Read Count: 255378
                                Read Latency: 0.962 ms.
                                Write Count: 393370
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 7007506
                                Compacted row mean size: 633

----------------
Keyspace: LH
                Read Count: 40729
                Read Latency: 1.1628788332637678 ms.
                Write Count: 265328
                Write Latency: 0.01778465898812036 ms.
                Pending Tasks: 0
                                Column Family: UserPrefrences
                                SSTable count: 6
                                Space used (live): 639728096
                                Space used (total): 786966376
                                Number of Keys (estimate): 2790912
                                Memtable Columns Count: 14546
                                Memtable Data Size: 7532750
                                Memtable Switch Count: 9
                                Read Count: 18874
                                Read Latency: 0.483 ms.
                                Write Count: 107364
                                Write Latency: 0.015 ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 87
                                Compacted row maximum size: 372
                                Compacted row mean size: 239

                                Column Family: LastViewedHotels
                                SSTable count: 4
                                Space used (live): 218367054
                                Space used (total): 227203324
                                Number of Keys (estimate): 1096192
                                Memtable Columns Count: 3724
                                Memtable Data Size: 5134228
                                Memtable Switch Count: 9
                                Read Count: 6085
                                Read Latency: 0.939 ms.
                                Write Count: 18849
                                Write Latency: 0.022 ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 73
                                Compacted row maximum size: 51012
                                Compacted row mean size: 186

                                Column Family: BookedHotels
                                SSTable count: 2
                                Space used (live): 8288377
                                Space used (total): 24867472
                                Number of Keys (estimate): 44928
                                Memtable Columns Count: 24
                                Memtable Data Size: 46264
                                Memtable Switch Count: 8
                                Read Count: 111
                                Read Latency: 0.025 ms.
                                Write Count: 209
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 87
                                Compacted row maximum size: 35425
                                Compacted row mean size: 152

                                Column Family: HotelMessage
                                SSTable count: 3
                                Space used (live): 683567
                                Space used (total): 2143936
                                Number of Keys (estimate): 1536
                                Memtable Columns Count: 14536
                                Memtable Data Size: 3764728
                                Memtable Switch Count: 8
                                Read Count: 4804
                                Read Latency: 2.714 ms.
                                Write Count: 64967
                                Write Latency: 0.012 ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 87
                                Compacted row maximum size: 20501
                                Compacted row mean size: 496

                                Column Family: SavedHotels
                                SSTable count: 3
                                Space used (live): 1035147
                                Space used (total): 3097633
                                Number of Keys (estimate): 5760
                                Memtable Columns Count: 8
                                Memtable Data Size: 15376
                                Memtable Switch Count: 7
                                Read Count: 3971
                                Read Latency: 0.023 ms.
                                Write Count: 73
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 2299
                                Compacted row mean size: 170

                                Column Family: SavedHotelsInverted
                                SSTable count: 1
                                Space used (live): 1022963
                                Space used (total): 4108255
                                Number of Keys (estimate): 5504
                                Memtable Columns Count: 8
                                Memtable Data Size: 450
                                Memtable Switch Count: 7
                                Read Count: 3
                                Read Latency: NaN ms.
                                Write Count: 73
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 2299
                                Compacted row mean size: 162

                                Column Family: LastViewedHotelsInverted
                                SSTable count: 4
                                Space used (live): 243688384
                                Space used (total): 362107541
                                Number of Keys (estimate): 1257216
                                Memtable Columns Count: 2917
                                Memtable Data Size: 4254486
                                Memtable Switch Count: 9
                                Read Count: 2454
                                Read Latency: 0.091 ms.
                                Write Count: 18857
                                Write Latency: 0.018 ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 73
                                Compacted row maximum size: 51012
                                Compacted row mean size: 170

                                Column Family: SavedSearchRequest
                                SSTable count: 7
                                Space used (live): 7125532732
                                Space used (total): 7266974627
                                Number of Keys (estimate): 1526144
                                Memtable Columns Count: 8854
                                Memtable Data Size: 23875680
                                Memtable Switch Count: 9
                                Read Count: 4427
                                Read Latency: 5.022 ms.
                                Write Count: 54938
                                Write Latency: 0.026 ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 925
                                Compacted row maximum size: 4055269
                                Compacted row mean size: 4912

                                Column Family: HotelTariffs
                                SSTable count: 1
                                Space used (live): 46306159
                                Space used (total): 46306159
                                Number of Keys (estimate): 7680
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 0
                                Read Count: 0
                                Read Latency: NaN ms.
                                Write Count: 0
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 180
                                Compacted row maximum size: 9887
                                Compacted row mean size: 6535




Regards,
Shubham Srivastava | Technical Lead - Technology Development

+91 124 439 5428   |  MakeMyTrip.com, 103 Udyog Vihar Phase 1, Gurgaon, Haryana - 122 016,
India


[http://www.mailmktg.makemytrip.com/signature/images/bulb.gif]What's new? My Trip Rewards
- An exclusive loyalty program for MakeMyTrip customers.<https://rewards.makemytrip.com/MTR>

[http://www.mailmktg.makemytrip.com/signature/images/MMT-signature-footer-V4.gif]<http://www.makemytrip.com/>

[http://www.mailmktg.makemytrip.com/signature/images/map-icon.gif]<http://www.makemytrip.com/support/gurgaon-travel-agent-office.php>
Office Map

[http://www.mailmktg.makemytrip.com/signature/images/facebook-icon.gif]<http://www.facebook.com/pages/MakeMyTrip-Deals/120740541030?ref=search&sid=100000077980239.1422657277..1>
Facebook

[http://www.mailmktg.makemytrip.com/signature/images/twitter-icon.gif]<http://twitter.com/makemytripdeals>
Twitter




Mime
View raw message