cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shubham <shubha...@gmail.com>
Subject Cassandra Compaction with flush_largest_memtables and ParNew and CMS logs frequently
Date Mon, 06 Feb 2012 11:21:56 GMT
Hi,

I am using Solandra : Solr(3.3)+Cassandra(0.8.6) and with a single shard and
total Docs close to 300k.

I have a 7 node cluster with RF:7 and Read CL:1 with read_repair_chance to
1.0. At times some of nodes like 3-4 goes for compaction and prints messages
like:

 Heap is 0.7523523533493616 full.  You may need to reduce memtable and/or
cache sizes.  Cassandra will now flush up to the two largest memtables to
free up memory.  Adjust flush_largest_memtables_at threshold in
cassandra.yaml if you don't want Cassandra to do this automatically
 WARN [ScheduledTasks:1] 2012-02-06 07:26:39,684 StorageService.java (line
2418) Flushing ColumnFamilyStore(table='LH', columnFamily='UserPrefrences')
to relieve memory pressure

INFO [ScheduledTasks:1] 2012-02-05 01:37:51,821 GCInspector.java (line 122)
GC for ConcurrentMarkSweep: 1234 ms for 1 collections, 2220779224 used; max
is 4080271360

 INFO [ScheduledTasks:1] 2012-02-05 01:38:20,615 GCInspector.java (line 122)
GC for ParNew: 644 ms for 1 collections, 1602357816 used; max is 4080271360
 INFO [ScheduledTasks:1] 2012-02-05 01:38:47,772 GCInspector.java (line 122)
GC for ParNew: 330 ms for 1 collections, 1756419088 used; max is 4080271360


GC stuff actually makes my nodes and infact the whole cluster pretty
unresponsive.This happens quite a while and most of the times after a write
job or so but not always after a write job.Memtable-threshold is 0.75.

There are only row_caches enabled as given the cfstats of LH keyspace.

*I have the whole setup which is majorly for reads.*

I have a 6Gb heap with the below settings.

MAX_HEAP_SIZE="6G"
HEAP_NEWSIZE="3G"
# GC tuning options
JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC"
#JVM_OPTS="$JVM_OPTS -server"
JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"
JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled"
JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=65"
JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly"

The memtable thresholds are the default ones. *Is the problem with the GC
and heap settings or the memtable thresholds*( Memtable thresholds:
1.1390624999999999/1440/243 (millions of ops/minutes/MB)).

How should I start tuning the properties as its killing my live env. Saw
through tpstats that lot of reads are getting dropped when this is
occurring.

Please help me urgently on this


I am sharing across the cfstats for all Cf's

 Keyspace: system
         Read Count: 19
         Read Latency: 5.6876842105263155 ms.
         Write Count: 2375
         Write Latency: 0.010477894736842104 ms.
         Pending Tasks: 0
                 Column Family: NodeIdInfo
                 SSTable count: 0
                 Space used (live): 0
                 Space used (total): 0
                 Number of Keys (estimate): 0
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 0
                 Read Count: 0
                 Read Latency: NaN ms.
                 Write Count: 0
                 Write Latency: NaN ms.
                 Pending Tasks: 0
                 Key cache capacity: 1
                 Key cache size: 0
                 Key cache hit rate: NaN
                 Row cache: disabled
                 Compacted row minimum size: 0
                 Compacted row maximum size: 0
                 Compacted row mean size: 0
 
                 Column Family: HintsColumnFamily
                 SSTable count: 2
                 Space used (live): 492064
                 Space used (total): 492064
                 Number of Keys (estimate): 256
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 3
                 Read Count: 10
                 Read Latency: 5.643 ms.
                 Write Count: 2372
                 Write Latency: 0.010 ms.
                 Pending Tasks: 0
                 Key cache capacity: 2
                 Key cache size: 2
                 Key cache hit rate: 0.42857142857142855
                 Row cache: disabled
                 Compacted row minimum size: 219343
                 Compacted row maximum size: 263210
                 Compacted row mean size: 263210
 
                 Column Family: Schema
                 SSTable count: 2
                 Space used (live): 20827
                 Space used (total): 20827
                 Number of Keys (estimate): 256
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 0
                 Read Count: 3
                 Read Latency: 4.253 ms.
                 Write Count: 0
                 Write Latency: NaN ms.
                 Pending Tasks: 0
                 Key cache capacity: 2
                 Key cache size: 2
                 Key cache hit rate: 0.0
                 Row cache: disabled
                 Compacted row minimum size: 104
                 Compacted row maximum size: 8239
                 Compacted row mean size: 3314
 
                 Column Family: Migrations
                 SSTable count: 2
                 Space used (live): 33180
                 Space used (total): 33180
                 Number of Keys (estimate): 256
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 0
                 Read Count: 0
                 Read Latency: NaN ms.
                 Write Count: 0
                 Write Latency: NaN ms.
                 Pending Tasks: 0
                 Key cache capacity: 2
                 Key cache size: 0
                 Key cache hit rate: NaN
                 Row cache: disabled
                 Compacted row minimum size: 9888
                 Compacted row maximum size: 17084
                 Compacted row mean size: 14474
 
                 Column Family: IndexInfo
                 SSTable count: 0
                 Space used (live): 0
                 Space used (total): 0
                 Number of Keys (estimate): 0
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 0
                 Read Count: 0
                 Read Latency: NaN ms.
                 Write Count: 0
                 Write Latency: NaN ms.
                 Pending Tasks: 0
                 Key cache capacity: 1
                 Key cache size: 0
                 Key cache hit rate: NaN
                 Row cache: disabled
                 Compacted row minimum size: 0
                 Compacted row maximum size: 0
                 Compacted row mean size: 0
 
                 Column Family: LocationInfo
                 SSTable count: 3
                 Space used (live): 15844
                 Space used (total): 15844
                 Number of Keys (estimate): 384
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 2
                 Read Count: 6
                 Read Latency: 6.479 ms.
                 Write Count: 3
                 Write Latency: 0.004 ms.
                 Pending Tasks: 0
                 Key cache capacity: 3
                 Key cache size: 3
                 Key cache hit rate: 0.3333333333333333
                 Row cache: disabled
                 Compacted row minimum size: 73
                 Compacted row maximum size: 310
                 Compacted row mean size: 126
 
 ----------------
 Keyspace: L
         Read Count: 535675
         Read Latency: 4.472875136976712 ms.
         Write Count: 282216
         Write Latency: 0.08156946806701251 ms.
         Pending Tasks: 0
                 Column Family: FC
                 SSTable count: 6
                 Space used (live): 444790202
                 Space used (total): 444790202
                 Number of Keys (estimate): 768
                 Memtable Columns Count: 93231
                 Memtable Data Size: 172790560
                 Memtable Switch Count: 1
                 Read Count: 6651
                 Read Latency: 62.041 ms.
                 Write Count: 94235
                 Write Latency: 0.026 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 104
                 Compacted row maximum size: 10090808
                 Compacted row mean size: 4256880
 
                 Column Family: Docs
                 SSTable count: 7
                 Space used (live): 1487268134
                 Space used (total): 1487268134
                 Number of Keys (estimate): 303744
                 Memtable Columns Count: 98040
                 Memtable Data Size: 37732086
                 Memtable Switch Count: 1
                 Read Count: 125986
                 Read Latency: 1.868 ms.
                 Write Count: 11900
                 Write Latency: 0.520 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 36
                 Compacted row maximum size: 51012
                 Compacted row mean size: 5547
 
                 Column Family: SI
                 SSTable count: 3
                 Space used (live): 136512555
                 Space used (total): 136512555
                 Number of Keys (estimate): 286848
                 Memtable Columns Count: 5344
                 Memtable Data Size: 21139811
                 Memtable Switch Count: 1
                 Read Count: 27642
                 Read Latency: 0.880 ms.
                 Write Count: 5815
                 Write Latency: 0.093 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 73
                 Compacted row maximum size: 30130992
                 Compacted row mean size: 5022190
 
                 Column Family: TL
                 SSTable count: 4
                 Space used (live): 314821524
                 Space used (total): 314821524
                 Number of Keys (estimate): 512
                 Memtable Columns Count: 83428
                 Memtable Data Size: 16069959
                 Memtable Switch Count: 1
                 Read Count: 10867
                 Read Latency: 17.949 ms.
                 Write Count: 5386
                 Write Latency: 0.578 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 12108971
                 Compacted row maximum size: 186563160
                 Compacted row mean size: 90195666
 
                 Column Family: TI
                 SSTable count: 6
                 Space used (live): 1451685937
                 Space used (total): 1451685937
                 Number of Keys (estimate): 3910144
                 Memtable Columns Count: 166006
                 Memtable Data Size: 184308808
                 Memtable Switch Count: 1
                 Read Count: 364529
                 Read Latency: 4.194 ms.
                 Write Count: 164880
                 Write Latency: 0.065 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 104
                 Compacted row maximum size: 7007506
                 Compacted row mean size: 18922
 
 ----------------
 Keyspace: LH
         Read Count: 61353
         Read Latency: 0.20730264208759147 ms.
         Write Count: 56289
         Write Latency: 0.13461228659240704 ms.
         Pending Tasks: 0
                 Column Family: UserPrefrences
                 SSTable count: 4
                 Space used (live): 335309522
                 Space used (total): 335309522
                 Number of Keys (estimate): 1513600
                 Memtable Columns Count: 12807
                 Memtable Data Size: 9144714
                 Memtable Switch Count: 1
                 Read Count: 20132
                 Read Latency: 0.201 ms.
                 Write Count: 27817
                 Write Latency: 0.009 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache: disabled
                 Compacted row minimum size: 61
                 Compacted row maximum size: 215
                 Compacted row mean size: 179
 
                 Column Family: LastViewedHotels
                 SSTable count: 3
                 Space used (live): 152316912
                 Space used (total): 152316912
                 Number of Keys (estimate): 767104
                 Memtable Columns Count: 3059
                 Memtable Data Size: 2906758
                 Memtable Switch Count: 1
                 Read Count: 9077
                 Read Latency: 0.180 ms.
                 Write Count: 3585
                 Write Latency: 0.018 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 565
                 Row cache hit rate: 0.7896882229811611
                 Compacted row minimum size: 36
                 Compacted row maximum size: 51012
                 Compacted row mean size: 144
 
                 Column Family: BookedHotels
                 SSTable count: 3
                 Space used (live): 7274700
                 Space used (total): 7274700
                 Number of Keys (estimate): 39680
                 Memtable Columns Count: 7
                 Memtable Data Size: 392
                 Memtable Switch Count: 1
                 Read Count: 139
                 Read Latency: 0.027 ms.
                 Write Count: 10
                 Write Latency: 0.008 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 4
                 Row cache hit rate: 0.2302158273381295
                 Compacted row minimum size: 87
                 Compacted row maximum size: 35425
                 Compacted row mean size: 139
 
                 Column Family: HotelMessage
                 SSTable count: 1
                 Space used (live): 349735
                 Space used (total): 349735
                 Number of Keys (estimate): 512
                 Memtable Columns Count: 19424
                 Memtable Data Size: 3408866
                 Memtable Switch Count: 1
                 Read Count: 4726
                 Read Latency: 0.059 ms.
                 Write Count: 9431
                 Write Latency: 0.739 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 144
                 Row cache hit rate: 0.968049090139653
                 Compacted row minimum size: 87
                 Compacted row maximum size: 24601
                 Compacted row mean size: 867
 
                 Column Family: SavedHotels
                 SSTable count: 1
                 Space used (live): 650153
                 Space used (total): 650153
                 Number of Keys (estimate): 3456
                 Memtable Columns Count: 13
                 Memtable Data Size: 728
                 Memtable Switch Count: 1
                 Read Count: 4282
                 Read Latency: 0.029 ms.
                 Write Count: 15
                 Write Latency: 0.014 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 13
                 Row cache hit rate: 0.06819243344231668
                 Compacted row minimum size: 104
                 Compacted row maximum size: 2299
                 Compacted row mean size: 160
 
                 Column Family: SavedHotelsInverted
                 SSTable count: 1
                 Space used (live): 646988
                 Space used (total): 646988
                 Number of Keys (estimate): 3456
                 Memtable Columns Count: 13
                 Memtable Data Size: 728
                 Memtable Switch Count: 1
                 Read Count: 13
                 Read Latency: 3.014 ms.
                 Write Count: 15
                 Write Latency: 0.007 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 2
                 Row cache hit rate: 0.15384615384615385
                 Compacted row minimum size: 104
                 Compacted row maximum size: 2299
                 Compacted row mean size: 160
 
                 Column Family: LastViewedHotelsInverted
                 SSTable count: 4
                 Space used (live): 147678370
                 Space used (total): 147678370
                 Number of Keys (estimate): 770048
                 Memtable Columns Count: 2486
                 Memtable Data Size: 2524930
                 Memtable Switch Count: 1
                 Read Count: 2667
                 Read Latency: 0.622 ms.
                 Write Count: 3590
                 Write Latency: 0.010 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 477
                 Row cache hit rate: 0.3820772403449569
                 Compacted row minimum size: 36
                 Compacted row maximum size: 51012
                 Compacted row mean size: 160
 
                 Column Family: SavedSearchRequest
                 SSTable count: 11
                 Space used (live): 4560932807
                 Space used (total): 4560932807
                 Number of Keys (estimate): 1022336
                 Memtable Columns Count: 7762
                 Memtable Data Size: 17281321
                 Memtable Switch Count: 1
                 Read Count: 20317
                 Read Latency: 0.242 ms.
                 Write Count: 11827
                 Write Latency: 0.022 ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 619
                 Row cache hit rate: 0.7583304621745336
                 Compacted row minimum size: 925
                 Compacted row maximum size: 1955666
                 Compacted row mean size: 5014
 
                 Column Family: HotelTariffs
                 SSTable count: 3
                 Space used (live): 42775204
                 Space used (total): 42775204
                 Number of Keys (estimate): 18176
                 Memtable Columns Count: 0
                 Memtable Data Size: 0
                 Memtable Switch Count: 0
                 Read Count: 0
                 Read Latency: NaN ms.
                 Write Count: 0
                 Write Latency: NaN ms.
                 Pending Tasks: 0
                 Key cache: disabled
                 Row cache capacity: 10000
                 Row cache size: 0
                 Row cache hit rate: NaN
                 Compacted row minimum size: 180
                 Compacted row maximum size: 9887
                 Compacted row mean size: 2179
 


-========


TPStats when nodes are down

Pool Name                    Active   Pending      Completed   Blocked  All
time blocked
ReadStage                        32      8137        2481612         0                
0
RequestResponseStage              4        50        3050938         0                
0
MutationStage                     1         1        2313537         0                
0
ReadRepairStage                   0         0         420629         0                
0
ReplicateOnWriteStage             0         0              0         0                
0
GossipStage                       0         0         236929         0                
0
AntiEntropyStage                  0         0              0         0                
0
MigrationStage                    0         0              0         0                
0
MemtablePostFlusher               0         0             80         0                
0
StreamStage                       0         0              0         0                
0
FlushWriter                       0         0             80         0                
9
MiscStage                         0         0              0         0                
0
FlushSorter                       0         0              0         0                
0
InternalResponseStage             0         0              0         0                
0
HintedHandoff                     1         1             15         0                
0

Message type           Dropped
RANGE_SLICE                 19
READ_REPAIR                  0
BINARY                       0
READ                    111404
MUTATION                589630
REQUEST_RESPONSE          1415



===================


Please let me know how could I at-least get this on a working state.

Regards,
Shubham











--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-Compaction-with-flush-largest-memtables-and-ParNew-and-CMS-logs-frequently-tp7258286p7258286.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Mime
View raw message