Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0F8E49C02 for ; Mon, 6 Feb 2012 11:22:27 +0000 (UTC) Received: (qmail 28439 invoked by uid 500); 6 Feb 2012 11:22:24 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 28325 invoked by uid 500); 6 Feb 2012 11:22:23 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 28317 invoked by uid 500); 6 Feb 2012 11:22:22 -0000 Delivered-To: apmail-incubator-cassandra-user@incubator.apache.org Received: (qmail 28314 invoked by uid 99); 6 Feb 2012 11:22:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Feb 2012 11:22:22 +0000 X-ASF-Spam-Status: No, hits=2.0 required=5.0 tests=SPF_NEUTRAL,SUBJ_OBFU_PUNCT_FEW,URI_HEX X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 216.139.236.26 is neither permitted nor denied by domain of shubham.k@gmail.com) Received: from [216.139.236.26] (HELO sam.nabble.com) (216.139.236.26) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Feb 2012 11:22:17 +0000 Received: from jim.nabble.com ([192.168.236.80]) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RuMdw-00024n-GJ for cassandra-user@incubator.apache.org; Mon, 06 Feb 2012 03:21:56 -0800 Date: Mon, 6 Feb 2012 03:21:56 -0800 (PST) From: shubham To: cassandra-user@incubator.apache.org Message-ID: <1328527316489-7258286.post@n2.nabble.com> Subject: Cassandra Compaction with flush_largest_memtables and ParNew and CMS logs frequently MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi, I am using Solandra : Solr(3.3)+Cassandra(0.8.6) and with a single shard and total Docs close to 300k. I have a 7 node cluster with RF:7 and Read CL:1 with read_repair_chance to 1.0. At times some of nodes like 3-4 goes for compaction and prints messages like: Heap is 0.7523523533493616 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically WARN [ScheduledTasks:1] 2012-02-06 07:26:39,684 StorageService.java (line 2418) Flushing ColumnFamilyStore(table='LH', columnFamily='UserPrefrences') to relieve memory pressure INFO [ScheduledTasks:1] 2012-02-05 01:37:51,821 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 1234 ms for 1 collections, 2220779224 used; max is 4080271360 INFO [ScheduledTasks:1] 2012-02-05 01:38:20,615 GCInspector.java (line 122) GC for ParNew: 644 ms for 1 collections, 1602357816 used; max is 4080271360 INFO [ScheduledTasks:1] 2012-02-05 01:38:47,772 GCInspector.java (line 122) GC for ParNew: 330 ms for 1 collections, 1756419088 used; max is 4080271360 GC stuff actually makes my nodes and infact the whole cluster pretty unresponsive.This happens quite a while and most of the times after a write job or so but not always after a write job.Memtable-threshold is 0.75. There are only row_caches enabled as given the cfstats of LH keyspace. *I have the whole setup which is majorly for reads.* I have a 6Gb heap with the below settings. MAX_HEAP_SIZE="6G" HEAP_NEWSIZE="3G" # GC tuning options JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC" #JVM_OPTS="$JVM_OPTS -server" JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC" JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled" JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8" JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1" JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=65" JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly" The memtable thresholds are the default ones. *Is the problem with the GC and heap settings or the memtable thresholds*( Memtable thresholds: 1.1390624999999999/1440/243 (millions of ops/minutes/MB)). How should I start tuning the properties as its killing my live env. Saw through tpstats that lot of reads are getting dropped when this is occurring. Please help me urgently on this I am sharing across the cfstats for all Cf's Keyspace: system Read Count: 19 Read Latency: 5.6876842105263155 ms. Write Count: 2375 Write Latency: 0.010477894736842104 ms. Pending Tasks: 0 Column Family: NodeIdInfo SSTable count: 0 Space used (live): 0 Space used (total): 0 Number of Keys (estimate): 0 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 0 Read Latency: NaN ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 1 Key cache size: 0 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 0 Compacted row maximum size: 0 Compacted row mean size: 0 Column Family: HintsColumnFamily SSTable count: 2 Space used (live): 492064 Space used (total): 492064 Number of Keys (estimate): 256 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 3 Read Count: 10 Read Latency: 5.643 ms. Write Count: 2372 Write Latency: 0.010 ms. Pending Tasks: 0 Key cache capacity: 2 Key cache size: 2 Key cache hit rate: 0.42857142857142855 Row cache: disabled Compacted row minimum size: 219343 Compacted row maximum size: 263210 Compacted row mean size: 263210 Column Family: Schema SSTable count: 2 Space used (live): 20827 Space used (total): 20827 Number of Keys (estimate): 256 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 3 Read Latency: 4.253 ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 2 Key cache size: 2 Key cache hit rate: 0.0 Row cache: disabled Compacted row minimum size: 104 Compacted row maximum size: 8239 Compacted row mean size: 3314 Column Family: Migrations SSTable count: 2 Space used (live): 33180 Space used (total): 33180 Number of Keys (estimate): 256 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 0 Read Latency: NaN ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 2 Key cache size: 0 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 9888 Compacted row maximum size: 17084 Compacted row mean size: 14474 Column Family: IndexInfo SSTable count: 0 Space used (live): 0 Space used (total): 0 Number of Keys (estimate): 0 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 0 Read Latency: NaN ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 1 Key cache size: 0 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 0 Compacted row maximum size: 0 Compacted row mean size: 0 Column Family: LocationInfo SSTable count: 3 Space used (live): 15844 Space used (total): 15844 Number of Keys (estimate): 384 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 2 Read Count: 6 Read Latency: 6.479 ms. Write Count: 3 Write Latency: 0.004 ms. Pending Tasks: 0 Key cache capacity: 3 Key cache size: 3 Key cache hit rate: 0.3333333333333333 Row cache: disabled Compacted row minimum size: 73 Compacted row maximum size: 310 Compacted row mean size: 126 ---------------- Keyspace: L Read Count: 535675 Read Latency: 4.472875136976712 ms. Write Count: 282216 Write Latency: 0.08156946806701251 ms. Pending Tasks: 0 Column Family: FC SSTable count: 6 Space used (live): 444790202 Space used (total): 444790202 Number of Keys (estimate): 768 Memtable Columns Count: 93231 Memtable Data Size: 172790560 Memtable Switch Count: 1 Read Count: 6651 Read Latency: 62.041 ms. Write Count: 94235 Write Latency: 0.026 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 104 Compacted row maximum size: 10090808 Compacted row mean size: 4256880 Column Family: Docs SSTable count: 7 Space used (live): 1487268134 Space used (total): 1487268134 Number of Keys (estimate): 303744 Memtable Columns Count: 98040 Memtable Data Size: 37732086 Memtable Switch Count: 1 Read Count: 125986 Read Latency: 1.868 ms. Write Count: 11900 Write Latency: 0.520 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 36 Compacted row maximum size: 51012 Compacted row mean size: 5547 Column Family: SI SSTable count: 3 Space used (live): 136512555 Space used (total): 136512555 Number of Keys (estimate): 286848 Memtable Columns Count: 5344 Memtable Data Size: 21139811 Memtable Switch Count: 1 Read Count: 27642 Read Latency: 0.880 ms. Write Count: 5815 Write Latency: 0.093 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 73 Compacted row maximum size: 30130992 Compacted row mean size: 5022190 Column Family: TL SSTable count: 4 Space used (live): 314821524 Space used (total): 314821524 Number of Keys (estimate): 512 Memtable Columns Count: 83428 Memtable Data Size: 16069959 Memtable Switch Count: 1 Read Count: 10867 Read Latency: 17.949 ms. Write Count: 5386 Write Latency: 0.578 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 12108971 Compacted row maximum size: 186563160 Compacted row mean size: 90195666 Column Family: TI SSTable count: 6 Space used (live): 1451685937 Space used (total): 1451685937 Number of Keys (estimate): 3910144 Memtable Columns Count: 166006 Memtable Data Size: 184308808 Memtable Switch Count: 1 Read Count: 364529 Read Latency: 4.194 ms. Write Count: 164880 Write Latency: 0.065 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 104 Compacted row maximum size: 7007506 Compacted row mean size: 18922 ---------------- Keyspace: LH Read Count: 61353 Read Latency: 0.20730264208759147 ms. Write Count: 56289 Write Latency: 0.13461228659240704 ms. Pending Tasks: 0 Column Family: UserPrefrences SSTable count: 4 Space used (live): 335309522 Space used (total): 335309522 Number of Keys (estimate): 1513600 Memtable Columns Count: 12807 Memtable Data Size: 9144714 Memtable Switch Count: 1 Read Count: 20132 Read Latency: 0.201 ms. Write Count: 27817 Write Latency: 0.009 ms. Pending Tasks: 0 Key cache: disabled Row cache: disabled Compacted row minimum size: 61 Compacted row maximum size: 215 Compacted row mean size: 179 Column Family: LastViewedHotels SSTable count: 3 Space used (live): 152316912 Space used (total): 152316912 Number of Keys (estimate): 767104 Memtable Columns Count: 3059 Memtable Data Size: 2906758 Memtable Switch Count: 1 Read Count: 9077 Read Latency: 0.180 ms. Write Count: 3585 Write Latency: 0.018 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 565 Row cache hit rate: 0.7896882229811611 Compacted row minimum size: 36 Compacted row maximum size: 51012 Compacted row mean size: 144 Column Family: BookedHotels SSTable count: 3 Space used (live): 7274700 Space used (total): 7274700 Number of Keys (estimate): 39680 Memtable Columns Count: 7 Memtable Data Size: 392 Memtable Switch Count: 1 Read Count: 139 Read Latency: 0.027 ms. Write Count: 10 Write Latency: 0.008 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 4 Row cache hit rate: 0.2302158273381295 Compacted row minimum size: 87 Compacted row maximum size: 35425 Compacted row mean size: 139 Column Family: HotelMessage SSTable count: 1 Space used (live): 349735 Space used (total): 349735 Number of Keys (estimate): 512 Memtable Columns Count: 19424 Memtable Data Size: 3408866 Memtable Switch Count: 1 Read Count: 4726 Read Latency: 0.059 ms. Write Count: 9431 Write Latency: 0.739 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 144 Row cache hit rate: 0.968049090139653 Compacted row minimum size: 87 Compacted row maximum size: 24601 Compacted row mean size: 867 Column Family: SavedHotels SSTable count: 1 Space used (live): 650153 Space used (total): 650153 Number of Keys (estimate): 3456 Memtable Columns Count: 13 Memtable Data Size: 728 Memtable Switch Count: 1 Read Count: 4282 Read Latency: 0.029 ms. Write Count: 15 Write Latency: 0.014 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 13 Row cache hit rate: 0.06819243344231668 Compacted row minimum size: 104 Compacted row maximum size: 2299 Compacted row mean size: 160 Column Family: SavedHotelsInverted SSTable count: 1 Space used (live): 646988 Space used (total): 646988 Number of Keys (estimate): 3456 Memtable Columns Count: 13 Memtable Data Size: 728 Memtable Switch Count: 1 Read Count: 13 Read Latency: 3.014 ms. Write Count: 15 Write Latency: 0.007 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 2 Row cache hit rate: 0.15384615384615385 Compacted row minimum size: 104 Compacted row maximum size: 2299 Compacted row mean size: 160 Column Family: LastViewedHotelsInverted SSTable count: 4 Space used (live): 147678370 Space used (total): 147678370 Number of Keys (estimate): 770048 Memtable Columns Count: 2486 Memtable Data Size: 2524930 Memtable Switch Count: 1 Read Count: 2667 Read Latency: 0.622 ms. Write Count: 3590 Write Latency: 0.010 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 477 Row cache hit rate: 0.3820772403449569 Compacted row minimum size: 36 Compacted row maximum size: 51012 Compacted row mean size: 160 Column Family: SavedSearchRequest SSTable count: 11 Space used (live): 4560932807 Space used (total): 4560932807 Number of Keys (estimate): 1022336 Memtable Columns Count: 7762 Memtable Data Size: 17281321 Memtable Switch Count: 1 Read Count: 20317 Read Latency: 0.242 ms. Write Count: 11827 Write Latency: 0.022 ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 619 Row cache hit rate: 0.7583304621745336 Compacted row minimum size: 925 Compacted row maximum size: 1955666 Compacted row mean size: 5014 Column Family: HotelTariffs SSTable count: 3 Space used (live): 42775204 Space used (total): 42775204 Number of Keys (estimate): 18176 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 0 Read Latency: NaN ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache: disabled Row cache capacity: 10000 Row cache size: 0 Row cache hit rate: NaN Compacted row minimum size: 180 Compacted row maximum size: 9887 Compacted row mean size: 2179 -======== TPStats when nodes are down Pool Name Active Pending Completed Blocked All time blocked ReadStage 32 8137 2481612 0 0 RequestResponseStage 4 50 3050938 0 0 MutationStage 1 1 2313537 0 0 ReadRepairStage 0 0 420629 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 236929 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage 0 0 0 0 0 MemtablePostFlusher 0 0 80 0 0 StreamStage 0 0 0 0 0 FlushWriter 0 0 80 0 9 MiscStage 0 0 0 0 0 FlushSorter 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 1 1 15 0 0 Message type Dropped RANGE_SLICE 19 READ_REPAIR 0 BINARY 0 READ 111404 MUTATION 589630 REQUEST_RESPONSE 1415 =================== Please let me know how could I at-least get this on a working state. Regards, Shubham -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-Compaction-with-flush-largest-memtables-and-ParNew-and-CMS-logs-frequently-tp7258286p7258286.html Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.