Tried all the possible options and nothing actually seems to work.

Was trying to get insights of where exactly problem is arising when writes on done on one node and read on another . I found that GC gets triggered when writes are done on the other node through RowMutationVerbHandler.

Settings

1.I tested this on two node setup with RF:2 and Read CL:1. 
2.Also heap is of 8G and Xmn:800M on both the nodes with 4Cores.
3.I am using concurrent_write:32 as default (8 * core). 
4. -Dcassandra.compaction.priority=1
5.No explicit GC settings commented the one in solandra-env.sh
6.in_memory_compaction_limit_in_mb: 1
7.read repair:0.1
8.concurrent_compactors: 1

Is there any way in which the writes can be made pretty slow on different nodes. Ideally I would like data to be written on one node and eventually replicating across other nodes I dont really need a real time update, so can pretty much live with slow writes.

Sharing the cassandra and GC logs as below

====================


DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf7365617263686669656c64efbfbf64657320', modifications=[ColumnFamily(TI [357025:false:4@1330861061454,])]) applied.  Sending response to 14541197@/10.86.29.21
DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])])
DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365
DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied.  Sending response to 14541198@/10.86.29.21
DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])])
DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573
DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied.  Sending response to 14541199@/10.86.29.21
DEBUG [248896865@qtp-1257398760-2] 2012-03-04 17:07:51,572 SolrIndexReader.java (line 928) getCoreCacheKey() - start
DEBUG [MutationStage:32] 2012-03-04 17:07:51,426 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf636f756e74727953636f7265', modifications=[ColumnFamily(FC [356996:false:4@1330861059172,])]) applied.  Sending response to 14532390@/10.86.29.21
DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:51,425 SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start
DEBUG [MutationStage:2] 2012-03-04 17:07:51,425 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6e616d65efbfbf6c61
DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:52,015 SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start
DEBUG [1861954021@qtp-1257398760-11] 2012-03-04 17:07:51,425 StorageProxy.java (line 696) Read: 0 ms.
 INFO [ScheduledTasks:1] 2012-03-04 17:07:51,425 GCInspector.java (line 123) GC for PS Scavenge: 443 ms for 1 collections, 5983756216 used; max is 8322088960
 
 
 
 =============
 
 DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf7365617263686669656c64efbfbf64657320', modifications=[ColumnFamily(TI [357025:false:4@1330861061454,])]) applied.  Sending response to 14541197@/10.86.29.21
 DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])])
 DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365
 DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied.  Sending response to 14541198@/10.86.29.21
 DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])])
 DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573
 DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied.  Sending response to 14541199@/10.86.29.21
 DEBUG [248896865@qtp-1257398760-2] 2012-03-04 17:07:51,572 SolrIndexReader.java (line 928) getCoreCacheKey() - start
 DEBUG [MutationStage:32] 2012-03-04 17:07:51,426 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf636f756e74727953636f7265', modifications=[ColumnFamily(FC [356996:false:4@1330861059172,])]) applied.  Sending response to 14532390@/10.86.29.21
 DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:51,425 SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start
 DEBUG [MutationStage:2] 2012-03-04 17:07:51,425 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6e616d65efbfbf6c61
 DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:52,015 SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start
 DEBUG [1861954021@qtp-1257398760-11] 2012-03-04 17:07:51,425 StorageProxy.java (line 696) Read: 0 ms.
  INFO [ScheduledTasks:1] 2012-03-04 17:07:51,425 GCInspector.java (line 123) GC for PS Scavenge: 443 ms for 1 collections, 5983756216 used; max is 8322088960



======================

DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf636f756e74727953636f7265efbfbfc2800000', modifications=[ColumnFamily(TI [357932:false:2@1330861128691,])])
DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf636f756e74727953636f7265efbfbfc2800000
DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf636f756e74727953636f7265efbfbfc2800000', modifications=[ColumnFamily(TI [357932:false:2@1330861128691,])]) applied.  Sending response to 14863674@/10.86.29.21
DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf6e756d4f664e6967687473efbfbf2001000000000000000000', modifications=[ColumnFamily(TI [357932:false:3@1330861128691,])])
DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6e756d4f664e6967687473efbfbf2001000000000000000000
DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf6e756d4f664e6967687473efbfbf2001000000000000000000', modifications=[ColumnFamily(TI [357932:false:3@1330861128691,])]) applied.  Sending response to 14863675@/10.86.29.21
DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf6164647231', modifications=[ColumnFamily(FC [357932:false:2@1330861128691,])])
DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6164647231
 INFO [ScheduledTasks:1] 2012-03-04 17:08:57,465 StatusLogger.java (line 65) MiscStage                         0         0         0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,899 StatusLogger.java (line 65) InternalResponseStage             0         0         0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 65) HintedHandoff                     0         0         0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 69) CompactionManager               n/a         0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 81) MessagingService                n/a     0,193
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 85) ColumnFamily                Memtable ops,data  Row cache size/cap  Key cache size/cap
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) system.NodeIdInfo                         0,0                 0/0                 0/1
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) system.IndexInfo                          0,0                 0/0                 0/1
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) system.LocationInfo                       0,0                 0/0                 3/3
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) system.Versions                         3,103                 0/0                 0/1
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) system.Migrations                         0,0                 0/0                 0/2
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) system.HintsColumnFamily                  0,0                 0/0                 0/1
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) system.Schema                             0,0                 0/0                 2/2
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) L.FC                         333959,109606980                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) L.TI                           26707,17617780                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) L.TL                          565188,54888553                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) L.SI                             5498,6575970                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) L.Docs                        260652,95271408                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) LH.SavedHotelsInverted                    0,0                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) LH.SavedHotels                            0,0                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.LastViewedHotelsInverted                1,56                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.SavedSearchRequest                  5,7308                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.HotelMessage                           0,0                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.UserPrefrences                      10,357                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.HotelTariffs                           0,0                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.LastViewedHotels                      1,56                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.BookedHotels                           0,0                 0/0                 0/0
 INFO [ScheduledTasks:1] 2012-03-04 17:08:58,903 GCInspector.java (line 123) GC for PS Scavenge: 859 ms for 3 collections, 2983178264 used; max is 8310358016
DEBUG [476953411@qtp-1257398760-13] 2012-03-04 17:08:58,918 SolrIndexReader.java (line 517) docFreq(Term) - end




=======================


DEBUG [MutationStage:30] 2012-03-04 17:09:01,473 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf7365617263686669656c64efbfbf6465207061726973', modifications=[ColumnFamily(TI [358128:false:4@1330861130798,])]) applied.  Sending response to 14871005@/10.86.29.21
DEBUG [MutationStage:30] 2012-03-04 17:09:01,473 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf6461746575706461746564efbfbf2001000000265d64493114', modifications=[ColumnFamily(TI [357958:true:4@1330861130944,])])
DEBUG [MutationStage:30] 2012-03-04 17:09:01,473 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6461746575706461746564efbfbf2001000000265d64493114
DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,387 SolrIndexReader.java (line 370) maxDoc() - start
DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,474 SolrIndexReader.java (line 375) maxDoc() - end
DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,474 SolrIndexReader.java (line 370) maxDoc() - start
DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,474 SolrIndexReader.java (line 375) maxDoc() - end
DEBUG [1435779426@qtp-1257398760-8] 2012-03-04 17:09:01,204 SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start
DEBUG [MutationStage:11] 2012-03-04 17:09:01,201 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf7365617263686669656c64efbfbf76696c6c652064
DEBUG [MutationStage:11] 2012-03-04 17:09:01,474 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf7365617263686669656c64efbfbf76696c6c652064', modifications=[ColumnFamily(TI [357955:false:4@1330861130682,])]) applied.  Sending response to 14870425@/10.86.29.21
DEBUG [MutationStage:28] 2012-03-04 17:09:01,190 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf6c61746c6e675f305f636f6f7264696e617465efbfbf400c021f2d5f', modifications=[ColumnFamily(TI [357971:false:4@1330861128965,])]) applied.  Sending response to 14865348@/10.86.29.21
DEBUG [MutationStage:1] 2012-03-04 17:09:01,190 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf3537363433
DEBUG [412581934@qtp-1257398760-16] 2012-03-04 17:09:01,127 CustomQueryComponent.java (line 116) prepare(ResponseBuilder) - end
DEBUG [7421394@qtp-1257398760-22] 2012-03-04 17:09:01,047 StorageProxy.java (line 696) Read: 1477 ms.
DEBUG [2030027307@qtp-1257398760-21] 2012-03-04 17:09:01,031 SolrIndexSearcher.java (line 2490) setDocSet(DocSet) - start
DEBUG [2030027307@qtp-1257398760-21] 2012-03-04 17:09:01,475 SolrIndexSearcher.java (line 2499) setDocSet(DocSet) - end
 INFO [ScheduledTasks:1] 2012-03-04 17:09:00,937 GCInspector.java (line 123) GC for PS Scavenge: 351 ms for 1 collections, 3051670552 used; max is 8329953280



==========

DEBUG [MutationStage:24] 2012-03-04 17:09:03,382 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6164647231efbfbf6465
DEBUG [MutationStage:24] 2012-03-04 17:09:03,382 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf6164647231efbfbf6465', modifications=[ColumnFamily(TI [358305:false:4@1330861132989,])]) applied.  Sending response to 14880403@/10.86.29.21
DEBUG [MutationStage:18] 2012-03-04 17:09:03,141 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf6e616d65efbfbf76616c656e74696e', modifications=[ColumnFamily(TI [358305:false:5@1330861132989,])]) applied.  Sending response to 14880326@/10.86.29.21
DEBUG [MutationStage:1] 2012-03-04 17:09:03,052 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf4661782f50686f746f636f7079696e67', modifications=[ColumnFamily(TI [358305:false:2@1330861132990,])]) applied.  Sending response to 14880195@/10.86.29.21
DEBUG [MutationStage:1] 2012-03-04 17:09:03,382 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf6c61746c6e675f315f636f6f7264696e617465efbfbf38177e4a374800', modifications=[ColumnFamily(TI [358252:true:4@1330861132886,])])
DEBUG [MutationStage:1] 2012-03-04 17:09:03,382 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6c61746c6e675f315f636f6f7264696e617465efbfbf38177e4a374800
DEBUG [MutationStage:11] 2012-03-04 17:09:03,037 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf5065747320416c6c6f776564', modifications=[ColumnFamily(TI [358252:true:4@1330861132886,])]) applied.  Sending response to 14880194@/10.86.29.21
DEBUG [728126393@qtp-1257398760-17] 2012-03-04 17:09:03,037 StorageProxy.java (line 696) Read: 1396 ms.
DEBUG [MutationStage:28] 2012-03-04 17:09:03,024 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf44697361626c656420466163696c6974696573', modifications=[ColumnFamily(TI [358252:true:4@1330861132886,])]) applied.  Sending response to 14880157@/10.86.29.21
DEBUG [MutationStage:28] 2012-03-04 17:09:03,384 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf67726f7373546172696666efbfbf2001000000000000000000', modifications=[ColumnFamily(TI [358305:false:3@1330861132989,])])
DEBUG [MutationStage:28] 2012-03-04 17:09:03,384 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf67726f7373546172696666efbfbf2001000000000000000000
DEBUG [MutationStage:23] 2012-03-04 17:09:03,012 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6665617475726573746174efbfbf4e
 INFO [ScheduledTasks:1] 2012-03-04 17:09:02,938 GCInspector.java (line 123) GC for PS Scavenge: 319 ms for 1 collections, 3148126656 used; max is 8329166848


===== GC Logs=====

1248.456: [GC
Desired survivor size 266403840 bytes, new threshold 1 (max 15)
 [PSYoungGen: 461285K->63226K(548736K)] 7589727K->7360609K(8118144K), 0.2978660 secs] [Times: user=1.15 sys=0.01, real=0.29 secs]
Total time for which application threads were stopped: 0.2985800 seconds
1250.024: [GC
Desired survivor size 253493248 bytes, new threshold 1 (max 15)
 [PSYoungGen: 341498K->76825K(571648K)] 7638881K->7433188K(8141056K), 0.1632840 secs] [Times: user=0.64 sys=0.00, real=0.16 secs]
1250.188: [Full GC [PSYoungGen: 76825K->0K(571648K)] [PSOldGen: 7356362K->2356764K(7569408K)] 7433188K->2356764K(8141056K) [PSPermGen: 32579K->32579K(61056K)], 8.1019330 secs] [Times: user=8.10 sys=0.00, real=8.10 secs]
Total time for which application threads were stopped: 8.2661710 seconds
Total time for which application threads were stopped: 0.0056460 seconds
1260.267: [GC
Desired survivor size 246808576 bytes, new threshold 1 (max 15)
 [PSYoungGen: 311488K->119239K(559040K)] 2668252K->2476003K(8128448K), 0.1174730 secs] [Times: user=0.42 sys=0.00, real=0.12 secs]
Total time for which application threads were stopped: 0.1181790 seconds
Total time for which application threads were stopped: 0.0071000 seconds
1262.020: [GC
Desired survivor size 247529472 bytes, new threshold 1 (max 15)
 [PSYoungGen: 430727K->138637K(576768K)] 2787491K->2612633K(8146176K), 0.2413110 secs] [Times: user=0.84 sys=0.00, real=0.25 secs]
Total time for which application threads were stopped: 0.2472850 seconds
Total time for which application threads were stopped: 0.0012250 seconds
1263.959: [GC
Desired survivor size 236191744 bytes, new threshold 1 (max 15)
 [PSYoungGen: 474381K->85226K(577472K)] 2948377K->2686551K(8146880K), 0.2765150 secs] [Times: user=1.06 sys=0.00, real=0.28 secs]
Total time for which application threads were stopped: 0.2811880 seconds
1265.567: [GC
Desired survivor size 257294336 bytes, new threshold 1 (max 15)
 [PSYoungGen: 420970K->191373K(547328K)] 3022295K->2873771K(8116736K), 0.2006350 secs] [Times: user=0.79 sys=0.00, real=0.20 secs]
Total time for which application threads were stopped: 0.2067200 seconds
Total time for which application threads were stopped: 0.0057330 seconds
1267.417: [GC
Desired survivor size 264765440 bytes, new threshold 1 (max 15)
 [PSYoungGen: 508045K->167618K(484352K)] 3190443K->3040491K(8053760K), 0.3643920 secs] [Times: user=1.25 sys=0.00, real=0.36 secs]
Total time for which application threads were stopped: 0.3704240 seconds
1268.904: [GC
Desired survivor size 252641280 bytes, new threshold 1 (max 15)
 [PSYoungGen: 484290K->100796K(572480K)] 3357163K->3140418K(8141888K), 0.3179310 secs] [Times: user=1.25 sys=0.00, real=0.31 secs]
Total time for which application threads were stopped: 0.3241000 seconds
Total time for which application threads were stopped: 0.0057010 seconds
1270.689: [GC
Desired survivor size 242483200 bytes, new threshold 1 (max 15)
 [PSYoungGen: 414716K->123054K(560640K)] 3454338K->3258021K(8130048K), 0.1692080 secs]

Regards,
Shubham


From: aaron morton [aaron@thelastpickle.com]
Sent: Thursday, February 23, 2012 2:00 AM
To: user@cassandra.apache.org
Subject: Re: Maximum Row Size in Cassandra : Potential Bottleneck

I'm not familiar with the solandra data model, but they are wide rows. 

In the past I've done the following to reduce GC pressure with wide rows where I thought GC was associated with compaction. It's heavy handed and may put pressure on the IO system. Once things are stabilised you may want to undo some of the changes.

* modify cassandra-env.sh to make sure the new heap is no more than 800M .
* modify cassandra.yaml and change concurrent_compactors to 2 
* modify cassandra.yaml and reduce in_memory_compaction_limit_in_mb to 32 

Then start watching the logs. 

Hope that helps. 

 

-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 21/02/2012, at 10:39 PM, Shubham Srivastava wrote:

Hi,
 
I am using cassandra0.8.6 for saving inverted indexes through Solr(Solr3.3 + Cassandra).
 
I am having problem while doing writes which causes  lot of GC activity where CPU usage increases + heap size usage and my nodes goes on a standstill state dropping most of reads and writes as per cfstats given below.
 
Looking at the row sizes of some of CF’s it seems to be on the higher side right from average row size of TL from 77 mb with max row size as 186 mb.Typically TL family is used for writing various terms of an indexed field and so on.
 
While doing writes to Cassandra we update/write back this column family + others as well. Is the row size like of TL normal or is it on the upper side.
 
My system can sustain small writes which actually happened after reducing the in in_memory_compaction_limit_in_mb: to 2 from 64 and read_repair_chance to 0.1 from 1.
 
I have a 7 node setup with RF:7 and Read CL:1 with 8 core processor and 14GB RAM. Hepa size is 8Gb on all the nodes.
 
 
 
 
CFSTATS:
 
Keyspace: system
                Read Count: 19
                Read Latency: 8.130157894736843 ms.
                Write Count: 31
                Write Latency: 0.03029032258064516 ms.
                Pending Tasks: 0
                                Column Family: NodeIdInfo
                                SSTable count: 1
                                Space used (live): 4516
                                Space used (total): 4516
                                Number of Keys (estimate): 128
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 0
                                Read Count: 0
                                Read Latency: NaN ms.
                                Write Count: 0
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 1
                                Key cache size: 0
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 87
                                Compacted row maximum size: 103
                                Compacted row mean size: 103
 
                                Column Family: HintsColumnFamily
                                SSTable count: 2
                                Space used (live): 9880
                                Space used (total): 34550
                                Number of Keys (estimate): 256
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 5
                                Read Count: 10
                                Read Latency: NaN ms.
                                Write Count: 28
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 2
                                Key cache size: 2
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 125
                                Compacted row maximum size: 149
                                Compacted row mean size: 149
 
                                Column Family: Schema
                                SSTable count: 1
                                Space used (live): 192414
                                Space used (total): 192414
                                Number of Keys (estimate): 128
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 0
                                Read Count: 3
                                Read Latency: NaN ms.
                                Write Count: 0
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 1
                                Key cache size: 1
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 8239
                                Compacted row mean size: 7794
 
                                Column Family: Migrations
                                SSTable count: 1
                                Space used (live): 300650
                                Space used (total): 300650
                                Number of Keys (estimate): 128
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 0
                                Read Count: 0
                                Read Latency: NaN ms.
                                Write Count: 0
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 1
                                Key cache size: 0
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 263211
                                Compacted row maximum size: 315852
                                Compacted row mean size: 315852
 
                                Column Family: IndexInfo
                                SSTable count: 0
                                Space used (live): 0
                                Space used (total): 0
                                Number of Keys (estimate): 0
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 0
                                Read Count: 0
                                Read Latency: NaN ms.
                                Write Count: 0
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 1
                                Key cache size: 0
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 0
                                Compacted row maximum size: 0
                                Compacted row mean size: 0
 
                                Column Family: LocationInfo
                                SSTable count: 2
                                Space used (live): 11496
                                Space used (total): 11496
                                Number of Keys (estimate): 256
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 2
                                Read Count: 6
                                Read Latency: NaN ms.
                                Write Count: 3
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache capacity: 2
                                Key cache size: 2
                                Key cache hit rate: NaN
                                Row cache: disabled
                                Compacted row minimum size: 73
                                Compacted row maximum size: 310
                                Compacted row mean size: 146
 
----------------
Keyspace: L
                Read Count: 343746
                Read Latency: 5.11407672525644 ms.
                Write Count: 541480
                Write Latency: 0.023806109182241266 ms.
                Pending Tasks: 0
                                Column Family: FC
                                SSTable count: 4
                                Space used (live): 353742759
                                Space used (total): 370460012
                                Number of Keys (estimate): 512
                                Memtable Columns Count: 64604
                                Memtable Data Size: 65810913
                                Memtable Switch Count: 4
                                Read Count: 93
                                Read Latency: NaN ms.
                                Write Count: 138640
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 10090808
                                Compacted row mean size: 2650773
 
                                Column Family: Docs
                                SSTable count: 9
                                Space used (live): 1645341386
                                Space used (total): 1726113393
                                Number of Keys (estimate): 356608
                                Memtable Columns Count: 65268
                                Memtable Data Size: 27459963
                                Memtable Switch Count: 4
                                Read Count: 74566
                                Read Latency: 1.995 ms.
                                Write Count: 4688
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 87
                                Compacted row maximum size: 51012
                                Compacted row mean size: 5557
 
                                Column Family: SI
                                SSTable count: 4
                                Space used (live): 88904425
                                Space used (total): 105431969
                                Number of Keys (estimate): 288512
                                Memtable Columns Count: 1141
                                Memtable Data Size: 767070
                                Memtable Switch Count: 4
                                Read Count: 8473
                                Read Latency: 0.615 ms.
                                Write Count: 2447
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 25109160
                                Compacted row mean size: 4694
 
                                Column Family: TL
                                SSTable count: 5
                                Space used (live): 334547547
                                Space used (total): 348394559
                                Number of Keys (estimate): 640
                                Memtable Columns Count: 90495
                                Memtable Data Size: 8602591
                                Memtable Switch Count: 3
                                Read Count: 5236
                                Read Latency: 30.199 ms.
                                Write Count: 2335
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 454827
                                Compacted row maximum size: 186563160
                                Compacted row mean size: 77287206
 
                                Column Family: TI
                                SSTable count: 10
                                Space used (live): 3056043044
                                Space used (total): 3084009723
                                Number of Keys (estimate): 8494720
                                Memtable Columns Count: 181003
                                Memtable Data Size: 212200530
                                Memtable Switch Count: 4
                                Read Count: 255378
                                Read Latency: 0.962 ms.
                                Write Count: 393370
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 7007506
                                Compacted row mean size: 633
 
----------------
Keyspace: LH
                Read Count: 40729
                Read Latency: 1.1628788332637678 ms.
                Write Count: 265328
                Write Latency: 0.01778465898812036 ms.
                Pending Tasks: 0
                                Column Family: UserPrefrences
                                SSTable count: 6
                                Space used (live): 639728096
                                Space used (total): 786966376
                                Number of Keys (estimate): 2790912
                                Memtable Columns Count: 14546
                                Memtable Data Size: 7532750
                                Memtable Switch Count: 9
                                Read Count: 18874
                                Read Latency: 0.483 ms.
                                Write Count: 107364
                                Write Latency: 0.015 ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 87
                                Compacted row maximum size: 372
                                Compacted row mean size: 239
 
                                Column Family: LastViewedHotels
                                SSTable count: 4
                                Space used (live): 218367054
                                Space used (total): 227203324
                                Number of Keys (estimate): 1096192
                                Memtable Columns Count: 3724
                                Memtable Data Size: 5134228
                                Memtable Switch Count: 9
                                Read Count: 6085
                                Read Latency: 0.939 ms.
                                Write Count: 18849
                                Write Latency: 0.022 ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 73
                                Compacted row maximum size: 51012
                                Compacted row mean size: 186
 
                                Column Family: BookedHotels
                                SSTable count: 2
                                Space used (live): 8288377
                                Space used (total): 24867472
                                Number of Keys (estimate): 44928
                                Memtable Columns Count: 24
                                Memtable Data Size: 46264
                                Memtable Switch Count: 8
                                Read Count: 111
                                Read Latency: 0.025 ms.
                                Write Count: 209
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 87
                                Compacted row maximum size: 35425
                                Compacted row mean size: 152
 
                                Column Family: HotelMessage
                                SSTable count: 3
                                Space used (live): 683567
                                Space used (total): 2143936
                                Number of Keys (estimate): 1536
                                Memtable Columns Count: 14536
                                Memtable Data Size: 3764728
                                Memtable Switch Count: 8
                                Read Count: 4804
                                Read Latency: 2.714 ms.
                                Write Count: 64967
                                Write Latency: 0.012 ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 87
                                Compacted row maximum size: 20501
                                Compacted row mean size: 496
 
                                Column Family: SavedHotels
                                SSTable count: 3
                                Space used (live): 1035147
                                Space used (total): 3097633
                                Number of Keys (estimate): 5760
                                Memtable Columns Count: 8
                                Memtable Data Size: 15376
                                Memtable Switch Count: 7
                                Read Count: 3971
                                Read Latency: 0.023 ms.
                                Write Count: 73
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 2299
                                Compacted row mean size: 170
 
                                Column Family: SavedHotelsInverted
                                SSTable count: 1
                                Space used (live): 1022963
                                Space used (total): 4108255
                                Number of Keys (estimate): 5504
                                Memtable Columns Count: 8
                                Memtable Data Size: 450
                                Memtable Switch Count: 7
                                Read Count: 3
                                Read Latency: NaN ms.
                                Write Count: 73
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 104
                                Compacted row maximum size: 2299
                                Compacted row mean size: 162
 
                                Column Family: LastViewedHotelsInverted
                                SSTable count: 4
                                Space used (live): 243688384
                                Space used (total): 362107541
                                Number of Keys (estimate): 1257216
                                Memtable Columns Count: 2917
                                Memtable Data Size: 4254486
                                Memtable Switch Count: 9
                                Read Count: 2454
                                Read Latency: 0.091 ms.
                                Write Count: 18857
                                Write Latency: 0.018 ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 73
                                Compacted row maximum size: 51012
                                Compacted row mean size: 170
 
                                Column Family: SavedSearchRequest
                                SSTable count: 7
                                Space used (live): 7125532732
                                Space used (total): 7266974627
                                Number of Keys (estimate): 1526144
                                Memtable Columns Count: 8854
                                Memtable Data Size: 23875680
                                Memtable Switch Count: 9
                                Read Count: 4427
                                Read Latency: 5.022 ms.
                                Write Count: 54938
                                Write Latency: 0.026 ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 925
                                Compacted row maximum size: 4055269
                                Compacted row mean size: 4912
 
                                Column Family: HotelTariffs
                                SSTable count: 1
                                Space used (live): 46306159
                                Space used (total): 46306159
                                Number of Keys (estimate): 7680
                                Memtable Columns Count: 0
                                Memtable Data Size: 0
                                Memtable Switch Count: 0
                                Read Count: 0
                                Read Latency: NaN ms.
                                Write Count: 0
                                Write Latency: NaN ms.
                                Pending Tasks: 0
                                Key cache: disabled
                                Row cache: disabled
                                Compacted row minimum size: 180
                                Compacted row maximum size: 9887
                                Compacted row mean size: 6535
 
 
 
 
Regards,
Shubham Srivastava | Technical Lead - Technology Development
+91 124 439 5428   |  MakeMyTrip.com, 103 Udyog Vihar Phase 1, Gurgaon, Haryana - 122 016, India
<image004.gif>
Office Map
<image005.gif>
Facebook