Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B1E769DA2 for ; Wed, 22 Feb 2012 20:31:29 +0000 (UTC) Received: (qmail 38014 invoked by uid 500); 22 Feb 2012 20:31:26 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 37982 invoked by uid 500); 22 Feb 2012 20:31:26 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 37971 invoked by uid 99); 22 Feb 2012 20:31:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Feb 2012 20:31:26 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a54.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Feb 2012 20:31:16 +0000 Received: from homiemail-a54.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a54.g.dreamhost.com (Postfix) with ESMTP id 614563A4058 for ; Wed, 22 Feb 2012 12:30:52 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=DKhG3+Ft+m o/4Khs3XN25FNr8JEd8lQVlmI9n2xX28A6zcdIFJFtN09cP9LsS75hM1ZYoUqKzc auqzXGeFk/LDlVJG8Dp0cM6Kha4IaJXc7YKEgGxM++A2um1cmLJU5VJOUpHoAwjZ oiJ51NzKSg8lr98z3+ybeXQ0JHY1ygZC4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=qbYgll1C8xv8qPfG N2NB7ICbBys=; b=BmYHJVRDExOiWWkOmH56U8OJqkmBTp/dw8T2V+R0RjdonQda 363WcI1LcWFxkrIWyoGO41SgzoVB4mCDqE+Cxvo/+4gzE7KNtocLgIufmvOJwsnj BQSMGVphrkfqJRRWD8fax7vCVPTxHnyic00gh0EV1DRzUHWEjkd2NcXrSl0= Received: from 202-126-206-138.vectorcommunications.net.nz (unknown [202.126.206.138]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a54.g.dreamhost.com (Postfix) with ESMTPSA id D56693A4057 for ; Wed, 22 Feb 2012 12:30:50 -0800 (PST) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: multipart/alternative; boundary="Apple-Mail=_41EDC809-136B-4F35-A188-94E6565D0792" Subject: Re: Maximum Row Size in Cassandra : Potential Bottleneck Date: Thu, 23 Feb 2012 09:30:49 +1300 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: <37B05A52-34FF-4CFD-A2E0-E4385DAE641F@thelastpickle.com> X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_41EDC809-136B-4F35-A188-94E6565D0792 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 I'm not familiar with the solandra data model, but they are wide rows.=20= In the past I've done the following to reduce GC pressure with wide rows = where I thought GC was associated with compaction. It's heavy handed and = may put pressure on the IO system. Once things are stabilised you may = want to undo some of the changes. * modify cassandra-env.sh to make sure the new heap is no more than 800M = . * modify cassandra.yaml and change concurrent_compactors to 2=20 * modify cassandra.yaml and reduce in_memory_compaction_limit_in_mb to = 32=20 Then start watching the logs.=20 Hope that helps.=20 =20 ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 21/02/2012, at 10:39 PM, Shubham Srivastava wrote: > Hi, > =20 > I am using cassandra0.8.6 for saving inverted indexes through = Solr(Solr3.3 + Cassandra). > =20 > I am having problem while doing writes which causes lot of GC = activity where CPU usage increases + heap size usage and my nodes goes = on a standstill state dropping most of reads and writes as per cfstats = given below. > =20 > Looking at the row sizes of some of CF=92s it seems to be on the = higher side right from average row size of TL from 77 mb with max row = size as 186 mb.Typically TL family is used for writing various terms of = an indexed field and so on. > =20 > While doing writes to Cassandra we update/write back this column = family + others as well. Is the row size like of TL normal or is it on = the upper side. > =20 > My system can sustain small writes which actually happened after = reducing the in in_memory_compaction_limit_in_mb: to 2 from 64 and = read_repair_chance to 0.1 from 1. > =20 > I have a 7 node setup with RF:7 and Read CL:1 with 8 core processor = and 14GB RAM. Hepa size is 8Gb on all the nodes. > =20 > =20 > =20 > =20 > CFSTATS: > =20 > Keyspace: system > Read Count: 19 > Read Latency: 8.130157894736843 ms. > Write Count: 31 > Write Latency: 0.03029032258064516 ms. > Pending Tasks: 0 > Column Family: NodeIdInfo > SSTable count: 1 > Space used (live): 4516 > Space used (total): 4516 > Number of Keys (estimate): 128 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 0 > Read Count: 0 > Read Latency: NaN ms. > Write Count: 0 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache capacity: 1 > Key cache size: 0 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 87 > Compacted row maximum size: 103 > Compacted row mean size: 103 > =20 > Column Family: HintsColumnFamily > SSTable count: 2 > Space used (live): 9880 > Space used (total): 34550 > Number of Keys (estimate): 256 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 5 > Read Count: 10 > Read Latency: NaN ms. > Write Count: 28 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache capacity: 2 > Key cache size: 2 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 125 > Compacted row maximum size: 149 > Compacted row mean size: 149 > =20 > Column Family: Schema > SSTable count: 1 > Space used (live): 192414 > Space used (total): 192414 > Number of Keys (estimate): 128 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 0 > Read Count: 3 > Read Latency: NaN ms. > Write Count: 0 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache capacity: 1 > Key cache size: 1 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 104 > Compacted row maximum size: 8239 > Compacted row mean size: 7794 > =20 > Column Family: Migrations > SSTable count: 1 > Space used (live): 300650 > Space used (total): 300650 > Number of Keys (estimate): 128 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 0 > Read Count: 0 > Read Latency: NaN ms. > Write Count: 0 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache capacity: 1 > Key cache size: 0 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 263211 > Compacted row maximum size: 315852 > Compacted row mean size: 315852 > =20 > Column Family: IndexInfo > SSTable count: 0 > Space used (live): 0 > Space used (total): 0 > Number of Keys (estimate): 0 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 0 > Read Count: 0 > Read Latency: NaN ms. > Write Count: 0 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache capacity: 1 > Key cache size: 0 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 0 > Compacted row maximum size: 0 > Compacted row mean size: 0 > =20 > Column Family: LocationInfo > SSTable count: 2 > Space used (live): 11496 > Space used (total): 11496 > Number of Keys (estimate): 256 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 2 > Read Count: 6 > Read Latency: NaN ms. > Write Count: 3 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache capacity: 2 > Key cache size: 2 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 73 > Compacted row maximum size: 310 > Compacted row mean size: 146 > =20 > ---------------- > Keyspace: L > Read Count: 343746 > Read Latency: 5.11407672525644 ms. > Write Count: 541480 > Write Latency: 0.023806109182241266 ms. > Pending Tasks: 0 > Column Family: FC > SSTable count: 4 > Space used (live): 353742759 > Space used (total): 370460012 > Number of Keys (estimate): 512 > Memtable Columns Count: 64604 > Memtable Data Size: 65810913 > Memtable Switch Count: 4 > Read Count: 93 > Read Latency: NaN ms. > Write Count: 138640 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 104 > Compacted row maximum size: 10090808 > Compacted row mean size: 2650773 > =20 > Column Family: Docs > SSTable count: 9 > Space used (live): 1645341386 > Space used (total): 1726113393 > Number of Keys (estimate): 356608 > Memtable Columns Count: 65268 > Memtable Data Size: 27459963 > Memtable Switch Count: 4 > Read Count: 74566 > Read Latency: 1.995 ms. > Write Count: 4688 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 87 > Compacted row maximum size: 51012 > Compacted row mean size: 5557 > =20 > Column Family: SI > SSTable count: 4 > Space used (live): 88904425 > Space used (total): 105431969 > Number of Keys (estimate): 288512 > Memtable Columns Count: 1141 > Memtable Data Size: 767070 > Memtable Switch Count: 4 > Read Count: 8473 > Read Latency: 0.615 ms. > Write Count: 2447 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 104 > Compacted row maximum size: 25109160 > Compacted row mean size: 4694 > =20 > Column Family: TL > SSTable count: 5 > Space used (live): 334547547 > Space used (total): 348394559 > Number of Keys (estimate): 640 > Memtable Columns Count: 90495 > Memtable Data Size: 8602591 > Memtable Switch Count: 3 > Read Count: 5236 > Read Latency: 30.199 ms. > Write Count: 2335 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 454827 > Compacted row maximum size: 186563160 > Compacted row mean size: 77287206 > =20 > Column Family: TI > SSTable count: 10 > Space used (live): 3056043044 > Space used (total): 3084009723 > Number of Keys (estimate): 8494720 > Memtable Columns Count: 181003 > Memtable Data Size: 212200530 > Memtable Switch Count: 4 > Read Count: 255378 > Read Latency: 0.962 ms. > Write Count: 393370 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 104 > Compacted row maximum size: 7007506 > Compacted row mean size: 633 > =20 > ---------------- > Keyspace: LH > Read Count: 40729 > Read Latency: 1.1628788332637678 ms. > Write Count: 265328 > Write Latency: 0.01778465898812036 ms. > Pending Tasks: 0 > Column Family: UserPrefrences > SSTable count: 6 > Space used (live): 639728096 > Space used (total): 786966376 > Number of Keys (estimate): 2790912 > Memtable Columns Count: 14546 > Memtable Data Size: 7532750 > Memtable Switch Count: 9 > Read Count: 18874 > Read Latency: 0.483 ms. > Write Count: 107364 > Write Latency: 0.015 ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 87 > Compacted row maximum size: 372 > Compacted row mean size: 239 > =20 > Column Family: LastViewedHotels > SSTable count: 4 > Space used (live): 218367054 > Space used (total): 227203324 > Number of Keys (estimate): 1096192 > Memtable Columns Count: 3724 > Memtable Data Size: 5134228 > Memtable Switch Count: 9 > Read Count: 6085 > Read Latency: 0.939 ms. > Write Count: 18849 > Write Latency: 0.022 ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 73 > Compacted row maximum size: 51012 > Compacted row mean size: 186 > =20 > Column Family: BookedHotels > SSTable count: 2 > Space used (live): 8288377 > Space used (total): 24867472 > Number of Keys (estimate): 44928 > Memtable Columns Count: 24 > Memtable Data Size: 46264 > Memtable Switch Count: 8 > Read Count: 111 > Read Latency: 0.025 ms. > Write Count: 209 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 87 > Compacted row maximum size: 35425 > Compacted row mean size: 152 > =20 > Column Family: HotelMessage > SSTable count: 3 > Space used (live): 683567 > Space used (total): 2143936 > Number of Keys (estimate): 1536 > Memtable Columns Count: 14536 > Memtable Data Size: 3764728 > Memtable Switch Count: 8 > Read Count: 4804 > Read Latency: 2.714 ms. > Write Count: 64967 > Write Latency: 0.012 ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 87 > Compacted row maximum size: 20501 > Compacted row mean size: 496 > =20 > Column Family: SavedHotels > SSTable count: 3 > Space used (live): 1035147 > Space used (total): 3097633 > Number of Keys (estimate): 5760 > Memtable Columns Count: 8 > Memtable Data Size: 15376 > Memtable Switch Count: 7 > Read Count: 3971 > Read Latency: 0.023 ms. > Write Count: 73 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 104 > Compacted row maximum size: 2299 > Compacted row mean size: 170 > =20 > Column Family: SavedHotelsInverted > SSTable count: 1 > Space used (live): 1022963 > Space used (total): 4108255 > Number of Keys (estimate): 5504 > Memtable Columns Count: 8 > Memtable Data Size: 450 > Memtable Switch Count: 7 > Read Count: 3 > Read Latency: NaN ms. > Write Count: 73 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 104 > Compacted row maximum size: 2299 > Compacted row mean size: 162 > =20 > Column Family: = LastViewedHotelsInverted > SSTable count: 4 > Space used (live): 243688384 > Space used (total): 362107541 > Number of Keys (estimate): 1257216 > Memtable Columns Count: 2917 > Memtable Data Size: 4254486 > Memtable Switch Count: 9 > Read Count: 2454 > Read Latency: 0.091 ms. > Write Count: 18857 > Write Latency: 0.018 ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 73 > Compacted row maximum size: 51012 > Compacted row mean size: 170 > =20 > Column Family: SavedSearchRequest > SSTable count: 7 > Space used (live): 7125532732 > Space used (total): 7266974627 > Number of Keys (estimate): 1526144 > Memtable Columns Count: 8854 > Memtable Data Size: 23875680 > Memtable Switch Count: 9 > Read Count: 4427 > Read Latency: 5.022 ms. > Write Count: 54938 > Write Latency: 0.026 ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 925 > Compacted row maximum size: 4055269 > Compacted row mean size: 4912 > =20 > Column Family: HotelTariffs > SSTable count: 1 > Space used (live): 46306159 > Space used (total): 46306159 > Number of Keys (estimate): 7680 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 0 > Read Count: 0 > Read Latency: NaN ms. > Write Count: 0 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache: disabled > Row cache: disabled > Compacted row minimum size: 180 > Compacted row maximum size: 9887 > Compacted row mean size: 6535 > =20 > =20 > =20 > =20 > Regards, > Shubham Srivastava | Technical Lead - Technology Development > +91 124 439 5428 | MakeMyTrip.com, 103 Udyog Vihar Phase 1, = Gurgaon, Haryana - 122 016, India > What's new? My Trip Rewards - An exclusive loyalty = program for MakeMyTrip customers. > > > Office Map > > Facebook > > Twitter > =20 --Apple-Mail=_41EDC809-136B-4F35-A188-94E6565D0792 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 I'm not familiar with the solandra data model, but = they are wide rows. 

In the past I've done the = following to reduce GC pressure with wide rows where I thought GC was = associated with compaction. It's heavy handed and may put pressure on = the IO system. Once things are stabilised you may want to undo some of = the changes.

* modify cassandra-env.sh to make = sure the new heap is no more than 800M .
* modify = cassandra.yaml and change concurrent_compactors to = 2 
* modify cassandra.yaml and = reduce in_memory_compaction_limit_in_mb to = 32 

Then start watching the = logs. 

Hope that = helps. 

 

http://www.thelastpickle.com

On 21/02/2012, at 10:39 PM, Shubham Srivastava = wrote:

Hi,
 
I am using cassandra0.8.6 for = saving inverted indexes through Solr(Solr3.3 + = Cassandra).
 
I am having problem while doing writes which causes =  lot of GC activity where CPU usage increases + heap size usage and = my nodes goes on a standstill state dropping most of reads and writes as = per cfstats given below.
 
Looking at the row sizes of some of CF=92s it seems to be = on the higher side right from average row size of TL from 77 mb with max = row size as 186 mb.Typically TL family is used for writing various terms = of an indexed field and so on.
While doing writes to Cassandra we = update/write back this column family + others as well. Is the row size = like of TL normal or is it on the upper side.
 
My system can sustain small = writes which actually happened after reducing the in = in_memory_compaction_limit_in_mb: to 2 from 64 and read_repair_chance to = 0.1 from 1.
 
I have a 7 node setup with RF:7 and Read CL:1 with 8 core = processor and 14GB RAM. Hepa size is 8Gb on all the = nodes.
 
 
 
CFSTATS:
 
Keyspace: = system
            =     Read Latency: 8.130157894736843 = ms.
            =     Write Latency: 0.03029032258064516 = ms.
            =             &n= bsp;       Column Family: = NodeIdInfo
 
 
 
            =             &n= bsp;       SSTable count: = 1
 
 
 
Keyspace: = L
            =     Read Latency: 5.11407672525644 = ms.
            =     Write Latency: 0.023806109182241266 = ms.
            =             &n= bsp;       Column Family: = FC
            =             &n= bsp;       Number of Keys (estimate): = 512
 
            =             &n= bsp;       Column Family: = Docs
 
 
 
            =             &n= bsp;       Column Family: = TI
 
Keyspace: = LH
            =     Read Latency: 1.1628788332637678 = ms.
            =     Write Latency: 0.01778465898812036 = ms.
            =             &n= bsp;       Column Family: = UserPrefrences
 
 
 
 
 
 
 
 
 
 
 
Regards,
Shubham Srivastava | 
+91 124 439 5428 =   |  MakeMyTrip.com, 103 Udyog Vihar Phase 1, Gurgaon, = Haryana - 122 016, India

<= /body>= --Apple-Mail=_41EDC809-136B-4F35-A188-94E6565D0792--