Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F10F59765 for ; Sat, 4 Feb 2012 04:59:36 +0000 (UTC) Received: (qmail 51177 invoked by uid 500); 4 Feb 2012 04:59:33 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 50671 invoked by uid 500); 4 Feb 2012 04:59:23 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 50662 invoked by uid 99); 4 Feb 2012 04:59:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Feb 2012 04:59:20 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Shubham.Srivastava@makemytrip.com designates 115.114.52.90 as permitted sender) Received: from [115.114.52.90] (HELO nsmail4.makemytrip.com) (115.114.52.90) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Feb 2012 04:59:12 +0000 Received: from mmt-srv-xfe2.mmt.com (unknown [172.16.8.162]) by nsmail4.makemytrip.com (Postfix) with ESMTP id B8110CF13C for ; Sat, 4 Feb 2012 10:28:49 +0530 (IST) Received: from MMT-SRV-XDB2.mmt.com ([fe80::655f:2e39:da9:5947]) by mmt-srv-xfe2.mmt.com ([fe80::1405:e804:6676:ecb3%17]) with mapi id 14.01.0339.001; Sat, 4 Feb 2012 10:28:49 +0530 From: Shubham Srivastava To: "user@cassandra.apache.org" Subject: Cassandra Compaction is Killing Thread-Topic: Cassandra Compaction is Killing Thread-Index: AQHM4vmtdgMm/FielkCtKJNy4DNt0w== Date: Sat, 4 Feb 2012 04:58:48 +0000 Message-ID: References: <1328331236.48734.ezmlm@cassandra.apache.org> In-Reply-To: <1328331236.48734.ezmlm@cassandra.apache.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-hashedpuzzle: BJsA BhMR CK9S DGWU DtgF Ewuf GXAo Hs/X J4Yk J+ao KSUV KZ9V K2fO LRKN LSe2 LuBh;1;dQBzAGUAcgBAAGMAYQBzAHMAYQBuAGQAcgBhAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{EDDD08EE-3601-440B-9931-3F473435B441};cwBoAHUAYgBoAGEAbQAuAHMAcgBpAHYAYQBzAHQAYQB2AGEAQABtAGEAawBlAG0AeQB0AHIAaQBwAC4AYwBvAG0A;Sat, 04 Feb 2012 04:57:39 GMT;QwBhAHMAcwBhAG4AZAByAGEAIABDAG8AbQBwAGEAYwB0AGkAbwBuACAAaQBzACAASwBpAGwAbABpAG4AZwA= x-cr-puzzleid: {EDDD08EE-3601-440B-9931-3F473435B441} x-originating-ip: [125.19.43.170] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 I have a cassandra setup with 7 Node Ring single DC with RF:7 and Read:CL:1= .There is live traffic on these nodes except one -> the traffic is 90% read= . There are also writes happening to all these nodes which typically are us= er specific data etc.These nodes aNow at times what is happening is there a= re 2-3 say nodes getting in a hanged state and eventually the whole ring be= haves so. There state in terms of CPU Usage is very High 17Loads ,network i= ncoming and outgoing requests packet were very high and there is continuous= GC(Major and Minor) pauses happening and Read Messages dropped simultaneo= usly.Also the SSTables couunt have increased and decreased for some familie= s during this time. So majorly its the GC and SStable compaction and Memtab= le flush happening. We are running cassandra behind solr using Solandra.The= max docs that we have are around 0.5Million. We keep one node as the main write node where we run sheduled job thats poo= ls data into the cluster from MySql db.These jobs are run hourly.Recently w= e have increased the data to somewhere to the size of 30X . Before the same= setup was stable and these jobs used to run in every 5 minutes apart from = the similar problem 2-3 times. GC:Settings 4Gb Heap Xmx,Xms 2Gb Young Geneartion: Xmn ParNew CMS -XX:+UseParNewGC"=20 -XX:+UseConcMarkSweepGC"=20 -XX:+CMSParallelRemarkEnabled"=20 -XX:SurvivorRatio=3D8"=20 -XX:MaxTenuringThreshold=3D1" -XX:CMSInitiatingOccupancyFraction=3D75" -XX:+UseCMSInitiatingOccupancyOnly" =20 Memtable Threshold is also:0.75 and rest default cassandra settings =20 Also we dont have in place a connectionpool for thrift.We are using cassan= dra:0.8.6 with solr:3.3 =20 Will share the Cfstats shortly or anything else needed for that matter.Can= you guys help me kn this. =20 =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 =20 Keyspace: system Read Count: 19 Read Latency: 5.6876842105263155 ms. Write Count: 2375 Write Latency: 0.010477894736842104 ms. Pending Tasks: 0 Column Family: NodeIdInfo SSTable count: 0 Space used (live): 0 Space used (total): 0 Number of Keys (estimate): 0 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 0 Read Latency: NaN ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 1 Key cache size: 0 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 0 Compacted row maximum size: 0 Compacted row mean size: 0 =20 Column Family: HintsColumnFamily SSTable count: 2 Space used (live): 492064 Space used (total): 492064 Number of Keys (estimate): 256 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 3 Read Count: 10 Read Latency: 5.643 ms. Write Count: 2372 Write Latency: 0.010 ms. Pending Tasks: 0 Key cache capacity: 2 Key cache size: 2 Key cache hit rate: 0.42857142857142855 Row cache: disabled Compacted row minimum size: 219343 Compacted row maximum size: 263210 Compacted row mean size: 263210 =20 Column Family: Schema SSTable count: 2 Space used (live): 20827 Space used (total): 20827 Number of Keys (estimate): 256 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 3 Read Latency: 4.253 ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 2 Key cache size: 2 Key cache hit rate: 0.0 Row cache: disabled Compacted row minimum size: 104 Compacted row maximum size: 8239 Compacted row mean size: 3314 =20 Column Family: Migrations SSTable count: 2 Space used (live): 33180 Space used (total): 33180 Number of Keys (estimate): 256 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 0 Read Latency: NaN ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 2 Key cache size: 0 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 9888 Compacted row maximum size: 17084 Compacted row mean size: 14474 =20 Column Family: IndexInfo SSTable count: 0 Space used (live): 0 Space used (total): 0 Number of Keys (estimate): 0 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 0 Read Latency: NaN ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 1 Key cache size: 0 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 0 Compacted row maximum size: 0 Compacted row mean size: 0 =20 Column Family: LocationInfo SSTable count: 3 Space used (live): 15844 Space used (total): 15844 Number of Keys (estimate): 384 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 2 Read Count: 6 Read Latency: 6.479 ms. Write Count: 3 Write Latency: 0.004 ms. Pending Tasks: 0 Key cache capacity: 3 Key cache size: 3 Key cache hit rate: 0.3333333333333333 Row cache: disabled Compacted row minimum size: 73 Compacted row maximum size: 310 Compacted row mean size: 126 =20 ---------------- Keyspace: L Read Count: 535675 Read Latency: 4.472875136976712 ms. Write Count: 282216 Write Latency: 0.08156946806701251 ms. Pending Tasks: 0 Column Family: FC SSTable count: 6 Space used (live): 444790202 Space used (total): 444790202 Number of Keys (estimate): 768 Memtable Columns Count: 93231 Memtable Data Size: 172790560 Memtable Switch Count: 1 Read Count: 6651 Read Latency: 62.041 ms. Write Count: 94235 Write Latency: 0.026 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 104 Compacted row maximum size: 10090808 Compacted row mean size: 4256880 =20 Column Family: Docs SSTable count: 7 Space used (live): 1487268134 Space used (total): 1487268134 Number of Keys (estimate): 303744 Memtable Columns Count: 98040 Memtable Data Size: 37732086 Memtable Switch Count: 1 Read Count: 125986 Read Latency: 1.868 ms. Write Count: 11900 Write Latency: 0.520 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 36 Compacted row maximum size: 51012 Compacted row mean size: 5547 =20 Column Family: SI SSTable count: 3 Space used (live): 136512555 Space used (total): 136512555 Number of Keys (estimate): 286848 Memtable Columns Count: 5344 Memtable Data Size: 21139811 Memtable Switch Count: 1 Read Count: 27642 Read Latency: 0.880 ms. Write Count: 5815 Write Latency: 0.093 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 73 Compacted row maximum size: 30130992 Compacted row mean size: 5022190 =20 Column Family: TL SSTable count: 4 Space used (live): 314821524 Space used (total): 314821524 Number of Keys (estimate): 512 Memtable Columns Count: 83428 Memtable Data Size: 16069959 Memtable Switch Count: 1 Read Count: 10867 Read Latency: 17.949 ms. Write Count: 5386 Write Latency: 0.578 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 12108971 Compacted row maximum size: 186563160 Compacted row mean size: 90195666 =20 Column Family: TI SSTable count: 6 Space used (live): 1451685937 Space used (total): 1451685937 Number of Keys (estimate): 3910144 Memtable Columns Count: 166006 Memtable Data Size: 184308808 Memtable Switch Count: 1 Read Count: 364529 Read Latency: 4.194 ms. Write Count: 164880 Write Latency: 0.065 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 104 Compacted row maximum size: 7007506 Compacted row mean size: 18922 =20 ---------------- Keyspace: LH Read Count: 61353 Read Latency: 0.20730264208759147 ms. Write Count: 56289 Write Latency: 0.13461228659240704 ms. Pending Tasks: 0 Column Family: UserPrefrences SSTable count: 4 Space used (live): 335309522 Space used (total): 335309522 Number of Keys (estimate): 1513600 Memtable Columns Count: 12807 Memtable Data Size: 9144714 Memtable Switch Count: 1 Read Count: 20132 Read Latency: 0.201 ms. Write Count: 27817 Write Latency: 0.009 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 61 Compacted row maximum size: 215 Compacted row mean size: 179 =20 Column Family: LastViewedHotels SSTable count: 3 Space used (live): 152316912 Space used (total): 152316912 Number of Keys (estimate): 767104 Memtable Columns Count: 3059 Memtable Data Size: 2906758 Memtable Switch Count: 1 Read Count: 9077 Read Latency: 0.180 ms. Write Count: 3585 Write Latency: 0.018 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 565 Row cache hit rate: 0.7896882229811611 Compacted row minimum size: 36 Compacted row maximum size: 51012 Compacted row mean size: 144 =20 Column Family: BookedHotels SSTable count: 3 Space used (live): 7274700 Space used (total): 7274700 Number of Keys (estimate): 39680 Memtable Columns Count: 7 Memtable Data Size: 392 Memtable Switch Count: 1 Read Count: 139 Read Latency: 0.027 ms. Write Count: 10 Write Latency: 0.008 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 4 Row cache hit rate: 0.2302158273381295 Compacted row minimum size: 87 Compacted row maximum size: 35425 Compacted row mean size: 139 =20 Column Family: HotelMessage SSTable count: 1 Space used (live): 349735 Space used (total): 349735 Number of Keys (estimate): 512 Memtable Columns Count: 19424 Memtable Data Size: 3408866 Memtable Switch Count: 1 Read Count: 4726 Read Latency: 0.059 ms. Write Count: 9431 Write Latency: 0.739 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 144 Row cache hit rate: 0.968049090139653 Compacted row minimum size: 87 Compacted row maximum size: 24601 Compacted row mean size: 867 =20 Column Family: SavedHotels SSTable count: 1 Space used (live): 650153 Space used (total): 650153 Number of Keys (estimate): 3456 Memtable Columns Count: 13 Memtable Data Size: 728 Memtable Switch Count: 1 Read Count: 4282 Read Latency: 0.029 ms. Write Count: 15 Write Latency: 0.014 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 13 Row cache hit rate: 0.06819243344231668 Compacted row minimum size: 104 Compacted row maximum size: 2299 Compacted row mean size: 160 =20 Column Family: SavedHotelsInverted SSTable count: 1 Space used (live): 646988 Space used (total): 646988 Number of Keys (estimate): 3456 Memtable Columns Count: 13 Memtable Data Size: 728 Memtable Switch Count: 1 Read Count: 13 Read Latency: 3.014 ms. Write Count: 15 Write Latency: 0.007 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 2 Row cache hit rate: 0.15384615384615385 Compacted row minimum size: 104 Compacted row maximum size: 2299 Compacted row mean size: 160 =20 Column Family: LastViewedHotelsInverted SSTable count: 4 Space used (live): 147678370 Space used (total): 147678370 Number of Keys (estimate): 770048 Memtable Columns Count: 2486 Memtable Data Size: 2524930 Memtable Switch Count: 1 Read Count: 2667 Read Latency: 0.622 ms. Write Count: 3590 Write Latency: 0.010 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 477 Row cache hit rate: 0.3820772403449569 Compacted row minimum size: 36 Compacted row maximum size: 51012 Compacted row mean size: 160 =20 Column Family: SavedSearchRequest SSTable count: 11 Space used (live): 4560932807 Space used (total): 4560932807 Number of Keys (estimate): 1022336 Memtable Columns Count: 7762 Memtable Data Size: 17281321 Memtable Switch Count: 1 Read Count: 20317 Read Latency: 0.242 ms. Write Count: 11827 Write Latency: 0.022 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 619 Row cache hit rate: 0.7583304621745336 Compacted row minimum size: 925 Compacted row maximum size: 1955666 Compacted row mean size: 5014 =20 Column Family: HotelTariffs SSTable count: 3 Space used (live): 42775204 Space used (total): 42775204 Number of Keys (estimate): 18176 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 0 Read Latency: NaN ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 0 Row cache hit rate: NaN Compacted row minimum size: 180 Compacted row maximum size: 9887 Compacted row mean size: 2179 I would appreciate a quick help. Regards, Shubham