incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Maximum Row Size in Cassandra : Potential Bottleneck
Date Mon, 05 Mar 2012 10:12:57 GMT
> Is there any way in which the writes can be made pretty slow on different nodes. Ideally I would like data to be written on one node and eventually replicating across other nodes I dont really need a real time update, so can pretty much live with slow writes.
Replicating "inside" the mutation request is a core feature of cassandra. 

You can hack something by disabling the the gossip on a node and doing the inserts on it (at CL One). Re-enable gossip and let HH send the data to the other nodes, or disable HH and use repair to distribute the changes. HH will be less resource intensive. 

> 1250.188: [Full GC [PSYoungGen: 76825K->0K(571648K)] [PSOldGen: 7356362K->2356764K(7569408K)] 7433188K->2356764K(8141056K) [PSPermGen: 32579K->32579K(61056K)], 8.1019330 secs] [Times: user=8.10 sys=0.00, real=8.10 secs]
What JVM are you using and what are the JVM options ? 
(The info I can find about PSYoungGen suggest it's pretty old. )

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 5/03/2012, at 1:13 AM, Shubham Srivastava wrote:

> Tried all the possible options and nothing actually seems to work.
> 
> Was trying to get insights of where exactly problem is arising when writes on done on one node and read on another . I found that GC gets triggered when writes are done on the other node through RowMutationVerbHandler.
> 
> Settings
> 
> 1.I tested this on two node setup with RF:2 and Read CL:1. 
> 2.Also heap is of 8G and Xmn:800M on both the nodes with 4Cores.
> 3.I am using concurrent_write:32 as default (8 * core). 
> 4. -Dcassandra.compaction.priority=1
> 5.No explicit GC settings commented the one in solandra-env.sh
> 6.in_memory_compaction_limit_in_mb: 1
> 7.read repair:0.1
> 8.concurrent_compactors: 1
> 
> 
> Is there any way in which the writes can be made pretty slow on different nodes. Ideally I would like data to be written on one node and eventually replicating across other nodes I dont really need a real time update, so can pretty much live with slow writes.
> 
> Sharing the cassandra and GC logs as below
> 
> ====================
> 
> 
> DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf7365617263686669656c64efbfbf64657320', modifications=[ColumnFamily(TI [357025:false:4@1330861061454,])]) applied.  Sending response to 14541197@/10.86.29.21
> DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])])
> DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365
> DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied.  Sending response to 14541198@/10.86.29.21
> DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])])
> DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573
> DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied.  Sending response to 14541199@/10.86.29.21
> DEBUG [248896865@qtp-1257398760-2] 2012-03-04 17:07:51,572 SolrIndexReader.java (line 928) getCoreCacheKey() - start
> DEBUG [MutationStage:32] 2012-03-04 17:07:51,426 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf636f756e74727953636f7265', modifications=[ColumnFamily(FC [356996:false:4@1330861059172,])]) applied.  Sending response to 14532390@/10.86.29.21
> DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:51,425 SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start
> DEBUG [MutationStage:2] 2012-03-04 17:07:51,425 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6e616d65efbfbf6c61
> DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:52,015 SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start
> DEBUG [1861954021@qtp-1257398760-11] 2012-03-04 17:07:51,425 StorageProxy.java (line 696) Read: 0 ms.
>  INFO [ScheduledTasks:1] 2012-03-04 17:07:51,425 GCInspector.java (line 123) GC for PS Scavenge: 443 ms for 1 collections, 5983756216 used; max is 8322088960
>  
>  
>  
>  =============
>  
>  DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf7365617263686669656c64efbfbf64657320', modifications=[ColumnFamily(TI [357025:false:4@1330861061454,])]) applied.  Sending response to 14541197@/10.86.29.21
>  DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])])
>  DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365
>  DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied.  Sending response to 14541198@/10.86.29.21
>  DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])])
>  DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573
>  DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied.  Sending response to 14541199@/10.86.29.21
>  DEBUG [248896865@qtp-1257398760-2] 2012-03-04 17:07:51,572 SolrIndexReader.java (line 928) getCoreCacheKey() - start
>  DEBUG [MutationStage:32] 2012-03-04 17:07:51,426 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf636f756e74727953636f7265', modifications=[ColumnFamily(FC [356996:false:4@1330861059172,])]) applied.  Sending response to 14532390@/10.86.29.21
>  DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:51,425 SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start
>  DEBUG [MutationStage:2] 2012-03-04 17:07:51,425 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6e616d65efbfbf6c61
>  DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:52,015 SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start
>  DEBUG [1861954021@qtp-1257398760-11] 2012-03-04 17:07:51,425 StorageProxy.java (line 696) Read: 0 ms.
>   INFO [ScheduledTasks:1] 2012-03-04 17:07:51,425 GCInspector.java (line 123) GC for PS Scavenge: 443 ms for 1 collections, 5983756216 used; max is 8322088960
> 
> 
> 
> ======================
> 
> DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf636f756e74727953636f7265efbfbfc2800000', modifications=[ColumnFamily(TI [357932:false:2@1330861128691,])])
> DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf636f756e74727953636f7265efbfbfc2800000
> DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf636f756e74727953636f7265efbfbfc2800000', modifications=[ColumnFamily(TI [357932:false:2@1330861128691,])]) applied.  Sending response to 14863674@/10.86.29.21
> DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf6e756d4f664e6967687473efbfbf2001000000000000000000', modifications=[ColumnFamily(TI [357932:false:3@1330861128691,])])
> DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6e756d4f664e6967687473efbfbf2001000000000000000000
> DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf6e756d4f664e6967687473efbfbf2001000000000000000000', modifications=[ColumnFamily(TI [357932:false:3@1330861128691,])]) applied.  Sending response to 14863675@/10.86.29.21
> DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf6164647231', modifications=[ColumnFamily(FC [357932:false:2@1330861128691,])])
> DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6164647231
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:57,465 StatusLogger.java (line 65) MiscStage                         0         0         0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,899 StatusLogger.java (line 65) InternalResponseStage             0         0         0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 65) HintedHandoff                     0         0         0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 69) CompactionManager               n/a         0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 81) MessagingService                n/a     0,193
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 85) ColumnFamily                Memtable ops,data  Row cache size/cap  Key cache size/cap
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) system.NodeIdInfo                         0,0                 0/0                 0/1
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) system.IndexInfo                          0,0                 0/0                 0/1
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) system.LocationInfo                       0,0                 0/0                 3/3
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) system.Versions                         3,103                 0/0                 0/1
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) system.Migrations                         0,0                 0/0                 0/2
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) system.HintsColumnFamily                  0,0                 0/0                 0/1
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) system.Schema                             0,0                 0/0                 2/2
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) L.FC                         333959,109606980                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) L.TI                           26707,17617780                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) L.TL                          565188,54888553                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) L.SI                             5498,6575970                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) L.Docs                        260652,95271408                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) LH.SavedHotelsInverted                    0,0                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) LH.SavedHotels                            0,0                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.LastViewedHotelsInverted                1,56                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.SavedSearchRequest                  5,7308                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.HotelMessage                           0,0                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.UserPrefrences                      10,357                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.HotelTariffs                           0,0                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.LastViewedHotels                      1,56                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) LH.BookedHotels                           0,0                 0/0                 0/0
>  INFO [ScheduledTasks:1] 2012-03-04 17:08:58,903 GCInspector.java (line 123) GC for PS Scavenge: 859 ms for 3 collections, 2983178264 used; max is 8310358016
> DEBUG [476953411@qtp-1257398760-13] 2012-03-04 17:08:58,918 SolrIndexReader.java (line 517) docFreq(Term) - end
> 
> 
> 
> 
> =======================
> 
> 
> DEBUG [MutationStage:30] 2012-03-04 17:09:01,473 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf7365617263686669656c64efbfbf6465207061726973', modifications=[ColumnFamily(TI [358128:false:4@1330861130798,])]) applied.  Sending response to 14871005@/10.86.29.21
> DEBUG [MutationStage:30] 2012-03-04 17:09:01,473 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf6461746575706461746564efbfbf2001000000265d64493114', modifications=[ColumnFamily(TI [357958:true:4@1330861130944,])])
> DEBUG [MutationStage:30] 2012-03-04 17:09:01,473 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6461746575706461746564efbfbf2001000000265d64493114
> DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,387 SolrIndexReader.java (line 370) maxDoc() - start
> DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,474 SolrIndexReader.java (line 375) maxDoc() - end
> DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,474 SolrIndexReader.java (line 370) maxDoc() - start
> DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,474 SolrIndexReader.java (line 375) maxDoc() - end
> DEBUG [1435779426@qtp-1257398760-8] 2012-03-04 17:09:01,204 SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start
> DEBUG [MutationStage:11] 2012-03-04 17:09:01,201 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf7365617263686669656c64efbfbf76696c6c652064
> DEBUG [MutationStage:11] 2012-03-04 17:09:01,474 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf7365617263686669656c64efbfbf76696c6c652064', modifications=[ColumnFamily(TI [357955:false:4@1330861130682,])]) applied.  Sending response to 14870425@/10.86.29.21
> DEBUG [MutationStage:28] 2012-03-04 17:09:01,190 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf6c61746c6e675f305f636f6f7264696e617465efbfbf400c021f2d5f', modifications=[ColumnFamily(TI [357971:false:4@1330861128965,])]) applied.  Sending response to 14865348@/10.86.29.21
> DEBUG [MutationStage:1] 2012-03-04 17:09:01,190 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf3537363433
> DEBUG [412581934@qtp-1257398760-16] 2012-03-04 17:09:01,127 CustomQueryComponent.java (line 116) prepare(ResponseBuilder) - end
> DEBUG [7421394@qtp-1257398760-22] 2012-03-04 17:09:01,047 StorageProxy.java (line 696) Read: 1477 ms.
> DEBUG [2030027307@qtp-1257398760-21] 2012-03-04 17:09:01,031 SolrIndexSearcher.java (line 2490) setDocSet(DocSet) - start
> DEBUG [2030027307@qtp-1257398760-21] 2012-03-04 17:09:01,475 SolrIndexSearcher.java (line 2499) setDocSet(DocSet) - end
>  INFO [ScheduledTasks:1] 2012-03-04 17:09:00,937 GCInspector.java (line 123) GC for PS Scavenge: 351 ms for 1 collections, 3051670552 used; max is 8329953280
> 
> 
> 
> ==========
> 
> DEBUG [MutationStage:24] 2012-03-04 17:09:03,382 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6164647231efbfbf6465
> DEBUG [MutationStage:24] 2012-03-04 17:09:03,382 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf6164647231efbfbf6465', modifications=[ColumnFamily(TI [358305:false:4@1330861132989,])]) applied.  Sending response to 14880403@/10.86.29.21
> DEBUG [MutationStage:18] 2012-03-04 17:09:03,141 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf6e616d65efbfbf76616c656e74696e', modifications=[ColumnFamily(TI [358305:false:5@1330861132989,])]) applied.  Sending response to 14880326@/10.86.29.21
> DEBUG [MutationStage:1] 2012-03-04 17:09:03,052 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf4661782f50686f746f636f7079696e67', modifications=[ColumnFamily(TI [358305:false:2@1330861132990,])]) applied.  Sending response to 14880195@/10.86.29.21
> DEBUG [MutationStage:1] 2012-03-04 17:09:03,382 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf6c61746c6e675f315f636f6f7264696e617465efbfbf38177e4a374800', modifications=[ColumnFamily(TI [358252:true:4@1330861132886,])])
> DEBUG [MutationStage:1] 2012-03-04 17:09:03,382 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6c61746c6e675f315f636f6f7264696e617465efbfbf38177e4a374800
> DEBUG [MutationStage:11] 2012-03-04 17:09:03,037 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf5065747320416c6c6f776564', modifications=[ColumnFamily(TI [358252:true:4@1330861132886,])]) applied.  Sending response to 14880194@/10.86.29.21
> DEBUG [728126393@qtp-1257398760-17] 2012-03-04 17:09:03,037 StorageProxy.java (line 696) Read: 1396 ms.
> DEBUG [MutationStage:28] 2012-03-04 17:09:03,024 RowMutationVerbHandler.java (line 56) RowMutation(keyspace='L', key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf44697361626c656420466163696c6974696573', modifications=[ColumnFamily(TI [358252:true:4@1330861132886,])]) applied.  Sending response to 14880157@/10.86.29.21
> DEBUG [MutationStage:28] 2012-03-04 17:09:03,384 RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='L', key='686f74656c737e30efbfbf67726f7373546172696666efbfbf2001000000000000000000', modifications=[ColumnFamily(TI [358305:false:3@1330861132989,])])
> DEBUG [MutationStage:28] 2012-03-04 17:09:03,384 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf67726f7373546172696666efbfbf2001000000000000000000
> DEBUG [MutationStage:23] 2012-03-04 17:09:03,012 Table.java (line 387) applying mutation of row 686f74656c737e30efbfbf6665617475726573746174efbfbf4e
>  INFO [ScheduledTasks:1] 2012-03-04 17:09:02,938 GCInspector.java (line 123) GC for PS Scavenge: 319 ms for 1 collections, 3148126656 used; max is 8329166848
> 
> 
> ===== GC Logs=====
> 
> 1248.456: [GC
> Desired survivor size 266403840 bytes, new threshold 1 (max 15)
>  [PSYoungGen: 461285K->63226K(548736K)] 7589727K->7360609K(8118144K), 0.2978660 secs] [Times: user=1.15 sys=0.01, real=0.29 secs]
> Total time for which application threads were stopped: 0.2985800 seconds
> 1250.024: [GC
> Desired survivor size 253493248 bytes, new threshold 1 (max 15)
>  [PSYoungGen: 341498K->76825K(571648K)] 7638881K->7433188K(8141056K), 0.1632840 secs] [Times: user=0.64 sys=0.00, real=0.16 secs]
> 1250.188: [Full GC [PSYoungGen: 76825K->0K(571648K)] [PSOldGen: 7356362K->2356764K(7569408K)] 7433188K->2356764K(8141056K) [PSPermGen: 32579K->32579K(61056K)], 8.1019330 secs] [Times: user=8.10 sys=0.00, real=8.10 secs]
> Total time for which application threads were stopped: 8.2661710 seconds
> Total time for which application threads were stopped: 0.0056460 seconds
> 1260.267: [GC
> Desired survivor size 246808576 bytes, new threshold 1 (max 15)
>  [PSYoungGen: 311488K->119239K(559040K)] 2668252K->2476003K(8128448K), 0.1174730 secs] [Times: user=0.42 sys=0.00, real=0.12 secs]
> Total time for which application threads were stopped: 0.1181790 seconds
> Total time for which application threads were stopped: 0.0071000 seconds
> 1262.020: [GC
> Desired survivor size 247529472 bytes, new threshold 1 (max 15)
>  [PSYoungGen: 430727K->138637K(576768K)] 2787491K->2612633K(8146176K), 0.2413110 secs] [Times: user=0.84 sys=0.00, real=0.25 secs]
> Total time for which application threads were stopped: 0.2472850 seconds
> Total time for which application threads were stopped: 0.0012250 seconds
> 1263.959: [GC
> Desired survivor size 236191744 bytes, new threshold 1 (max 15)
>  [PSYoungGen: 474381K->85226K(577472K)] 2948377K->2686551K(8146880K), 0.2765150 secs] [Times: user=1.06 sys=0.00, real=0.28 secs]
> Total time for which application threads were stopped: 0.2811880 seconds
> 1265.567: [GC
> Desired survivor size 257294336 bytes, new threshold 1 (max 15)
>  [PSYoungGen: 420970K->191373K(547328K)] 3022295K->2873771K(8116736K), 0.2006350 secs] [Times: user=0.79 sys=0.00, real=0.20 secs]
> Total time for which application threads were stopped: 0.2067200 seconds
> Total time for which application threads were stopped: 0.0057330 seconds
> 1267.417: [GC
> Desired survivor size 264765440 bytes, new threshold 1 (max 15)
>  [PSYoungGen: 508045K->167618K(484352K)] 3190443K->3040491K(8053760K), 0.3643920 secs] [Times: user=1.25 sys=0.00, real=0.36 secs]
> Total time for which application threads were stopped: 0.3704240 seconds
> 1268.904: [GC
> Desired survivor size 252641280 bytes, new threshold 1 (max 15)
>  [PSYoungGen: 484290K->100796K(572480K)] 3357163K->3140418K(8141888K), 0.3179310 secs] [Times: user=1.25 sys=0.00, real=0.31 secs]
> Total time for which application threads were stopped: 0.3241000 seconds
> Total time for which application threads were stopped: 0.0057010 seconds
> 1270.689: [GC
> Desired survivor size 242483200 bytes, new threshold 1 (max 15)
>  [PSYoungGen: 414716K->123054K(560640K)] 3454338K->3258021K(8130048K), 0.1692080 secs]
> 
> Regards,
> Shubham
> 
> 
> From: aaron morton [aaron@thelastpickle.com]
> Sent: Thursday, February 23, 2012 2:00 AM
> To: user@cassandra.apache.org
> Subject: Re: Maximum Row Size in Cassandra : Potential Bottleneck
> 
> I'm not familiar with the solandra data model, but they are wide rows. 
> 
> In the past I've done the following to reduce GC pressure with wide rows where I thought GC was associated with compaction. It's heavy handed and may put pressure on the IO system. Once things are stabilised you may want to undo some of the changes.
> 
> * modify cassandra-env.sh to make sure the new heap is no more than 800M .
> * modify cassandra.yaml and change concurrent_compactors to 2 
> * modify cassandra.yaml and reduce in_memory_compaction_limit_in_mb to 32 
> 
> Then start watching the logs. 
> 
> Hope that helps. 
> 
>  
> 
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 21/02/2012, at 10:39 PM, Shubham Srivastava wrote:
> 
>> Hi,
>>  
>> I am using cassandra0.8.6 for saving inverted indexes through Solr(Solr3.3 + Cassandra).
>>  
>> I am having problem while doing writes which causes  lot of GC activity where CPU usage increases + heap size usage and my nodes goes on a standstill state dropping most of reads and writes as per cfstats given below.
>>  
>> Looking at the row sizes of some of CF’s it seems to be on the higher side right from average row size of TL from 77 mb with max row size as 186 mb.Typically TL family is used for writing various terms of an indexed field and so on.
>>  
>> While doing writes to Cassandra we update/write back this column family + others as well. Is the row size like of TL normal or is it on the upper side.
>>  
>> My system can sustain small writes which actually happened after reducing the in in_memory_compaction_limit_in_mb: to 2 from 64 and read_repair_chance to 0.1 from 1.
>>  
>> I have a 7 node setup with RF:7 and Read CL:1 with 8 core processor and 14GB RAM. Hepa size is 8Gb on all the nodes.
>>  
>>  
>>  
>>  
>> CFSTATS:
>>  
>> Keyspace: system
>>                 Read Count: 19
>>                 Read Latency: 8.130157894736843 ms.
>>                 Write Count: 31
>>                 Write Latency: 0.03029032258064516 ms.
>>                 Pending Tasks: 0
>>                                 Column Family: NodeIdInfo
>>                                 SSTable count: 1
>>                                 Space used (live): 4516
>>                                 Space used (total): 4516
>>                                 Number of Keys (estimate): 128
>>                                 Memtable Columns Count: 0
>>                                 Memtable Data Size: 0
>>                                 Memtable Switch Count: 0
>>                                 Read Count: 0
>>                                 Read Latency: NaN ms.
>>                                 Write Count: 0
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache capacity: 1
>>                                 Key cache size: 0
>>                                 Key cache hit rate: NaN
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 87
>>                                 Compacted row maximum size: 103
>>                                 Compacted row mean size: 103
>>  
>>                                 Column Family: HintsColumnFamily
>>                                 SSTable count: 2
>>                                 Space used (live): 9880
>>                                 Space used (total): 34550
>>                                 Number of Keys (estimate): 256
>>                                 Memtable Columns Count: 0
>>                                 Memtable Data Size: 0
>>                                 Memtable Switch Count: 5
>>                                 Read Count: 10
>>                                 Read Latency: NaN ms.
>>                                 Write Count: 28
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache capacity: 2
>>                                 Key cache size: 2
>>                                 Key cache hit rate: NaN
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 125
>>                                 Compacted row maximum size: 149
>>                                 Compacted row mean size: 149
>>  
>>                                 Column Family: Schema
>>                                 SSTable count: 1
>>                                 Space used (live): 192414
>>                                 Space used (total): 192414
>>                                 Number of Keys (estimate): 128
>>                                 Memtable Columns Count: 0
>>                                 Memtable Data Size: 0
>>                                 Memtable Switch Count: 0
>>                                 Read Count: 3
>>                                 Read Latency: NaN ms.
>>                                 Write Count: 0
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache capacity: 1
>>                                 Key cache size: 1
>>                                 Key cache hit rate: NaN
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 104
>>                                 Compacted row maximum size: 8239
>>                                 Compacted row mean size: 7794
>>  
>>                                 Column Family: Migrations
>>                                 SSTable count: 1
>>                                 Space used (live): 300650
>>                                 Space used (total): 300650
>>                                 Number of Keys (estimate): 128
>>                                 Memtable Columns Count: 0
>>                                 Memtable Data Size: 0
>>                                 Memtable Switch Count: 0
>>                                 Read Count: 0
>>                                 Read Latency: NaN ms.
>>                                 Write Count: 0
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache capacity: 1
>>                                 Key cache size: 0
>>                                 Key cache hit rate: NaN
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 263211
>>                                 Compacted row maximum size: 315852
>>                                 Compacted row mean size: 315852
>>  
>>                                 Column Family: IndexInfo
>>                                 SSTable count: 0
>>                                 Space used (live): 0
>>                                 Space used (total): 0
>>                                 Number of Keys (estimate): 0
>>                                 Memtable Columns Count: 0
>>                                 Memtable Data Size: 0
>>                                 Memtable Switch Count: 0
>>                                 Read Count: 0
>>                                 Read Latency: NaN ms.
>>                                 Write Count: 0
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache capacity: 1
>>                                 Key cache size: 0
>>                                 Key cache hit rate: NaN
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 0
>>                                 Compacted row maximum size: 0
>>                                 Compacted row mean size: 0
>>  
>>                                 Column Family: LocationInfo
>>                                 SSTable count: 2
>>                                 Space used (live): 11496
>>                                 Space used (total): 11496
>>                                 Number of Keys (estimate): 256
>>                                 Memtable Columns Count: 0
>>                                 Memtable Data Size: 0
>>                                 Memtable Switch Count: 2
>>                                 Read Count: 6
>>                                 Read Latency: NaN ms.
>>                                 Write Count: 3
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache capacity: 2
>>                                 Key cache size: 2
>>                                 Key cache hit rate: NaN
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 73
>>                                 Compacted row maximum size: 310
>>                                 Compacted row mean size: 146
>>  
>> ----------------
>> Keyspace: L
>>                 Read Count: 343746
>>                 Read Latency: 5.11407672525644 ms.
>>                 Write Count: 541480
>>                 Write Latency: 0.023806109182241266 ms.
>>                 Pending Tasks: 0
>>                                 Column Family: FC
>>                                 SSTable count: 4
>>                                 Space used (live): 353742759
>>                                 Space used (total): 370460012
>>                                 Number of Keys (estimate): 512
>>                                 Memtable Columns Count: 64604
>>                                 Memtable Data Size: 65810913
>>                                 Memtable Switch Count: 4
>>                                 Read Count: 93
>>                                 Read Latency: NaN ms.
>>                                 Write Count: 138640
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 104
>>                                 Compacted row maximum size: 10090808
>>                                 Compacted row mean size: 2650773
>>  
>>                                 Column Family: Docs
>>                                 SSTable count: 9
>>                                 Space used (live): 1645341386
>>                                 Space used (total): 1726113393
>>                                 Number of Keys (estimate): 356608
>>                                 Memtable Columns Count: 65268
>>                                 Memtable Data Size: 27459963
>>                                 Memtable Switch Count: 4
>>                                 Read Count: 74566
>>                                 Read Latency: 1.995 ms.
>>                                 Write Count: 4688
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 87
>>                                 Compacted row maximum size: 51012
>>                                 Compacted row mean size: 5557
>>  
>>                                 Column Family: SI
>>                                 SSTable count: 4
>>                                 Space used (live): 88904425
>>                                 Space used (total): 105431969
>>                                 Number of Keys (estimate): 288512
>>                                 Memtable Columns Count: 1141
>>                                 Memtable Data Size: 767070
>>                                 Memtable Switch Count: 4
>>                                 Read Count: 8473
>>                                 Read Latency: 0.615 ms.
>>                                 Write Count: 2447
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 104
>>                                 Compacted row maximum size: 25109160
>>                                 Compacted row mean size: 4694
>>  
>>                                 Column Family: TL
>>                                 SSTable count: 5
>>                                 Space used (live): 334547547
>>                                 Space used (total): 348394559
>>                                 Number of Keys (estimate): 640
>>                                 Memtable Columns Count: 90495
>>                                 Memtable Data Size: 8602591
>>                                 Memtable Switch Count: 3
>>                                 Read Count: 5236
>>                                 Read Latency: 30.199 ms.
>>                                 Write Count: 2335
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 454827
>>                                 Compacted row maximum size: 186563160
>>                                 Compacted row mean size: 77287206
>>  
>>                                 Column Family: TI
>>                                 SSTable count: 10
>>                                 Space used (live): 3056043044
>>                                 Space used (total): 3084009723
>>                                 Number of Keys (estimate): 8494720
>>                                 Memtable Columns Count: 181003
>>                                 Memtable Data Size: 212200530
>>                                 Memtable Switch Count: 4
>>                                 Read Count: 255378
>>                                 Read Latency: 0.962 ms.
>>                                 Write Count: 393370
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 104
>>                                 Compacted row maximum size: 7007506
>>                                 Compacted row mean size: 633
>>  
>> ----------------
>> Keyspace: LH
>>                 Read Count: 40729
>>                 Read Latency: 1.1628788332637678 ms.
>>                 Write Count: 265328
>>                 Write Latency: 0.01778465898812036 ms.
>>                 Pending Tasks: 0
>>                                 Column Family: UserPrefrences
>>                                 SSTable count: 6
>>                                 Space used (live): 639728096
>>                                 Space used (total): 786966376
>>                                 Number of Keys (estimate): 2790912
>>                                 Memtable Columns Count: 14546
>>                                 Memtable Data Size: 7532750
>>                                 Memtable Switch Count: 9
>>                                 Read Count: 18874
>>                                 Read Latency: 0.483 ms.
>>                                 Write Count: 107364
>>                                 Write Latency: 0.015 ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 87
>>                                 Compacted row maximum size: 372
>>                                 Compacted row mean size: 239
>>  
>>                                 Column Family: LastViewedHotels
>>                                 SSTable count: 4
>>                                 Space used (live): 218367054
>>                                 Space used (total): 227203324
>>                                 Number of Keys (estimate): 1096192
>>                                 Memtable Columns Count: 3724
>>                                 Memtable Data Size: 5134228
>>                                 Memtable Switch Count: 9
>>                                 Read Count: 6085
>>                                 Read Latency: 0.939 ms.
>>                                 Write Count: 18849
>>                                 Write Latency: 0.022 ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 73
>>                                 Compacted row maximum size: 51012
>>                                 Compacted row mean size: 186
>>  
>>                                 Column Family: BookedHotels
>>                                 SSTable count: 2
>>                                 Space used (live): 8288377
>>                                 Space used (total): 24867472
>>                                 Number of Keys (estimate): 44928
>>                                 Memtable Columns Count: 24
>>                                 Memtable Data Size: 46264
>>                                 Memtable Switch Count: 8
>>                                 Read Count: 111
>>                                 Read Latency: 0.025 ms.
>>                                 Write Count: 209
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 87
>>                                 Compacted row maximum size: 35425
>>                                 Compacted row mean size: 152
>>  
>>                                 Column Family: HotelMessage
>>                                 SSTable count: 3
>>                                 Space used (live): 683567
>>                                 Space used (total): 2143936
>>                                 Number of Keys (estimate): 1536
>>                                 Memtable Columns Count: 14536
>>                                 Memtable Data Size: 3764728
>>                                 Memtable Switch Count: 8
>>                                 Read Count: 4804
>>                                 Read Latency: 2.714 ms.
>>                                 Write Count: 64967
>>                                 Write Latency: 0.012 ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 87
>>                                 Compacted row maximum size: 20501
>>                                 Compacted row mean size: 496
>>  
>>                                 Column Family: SavedHotels
>>                                 SSTable count: 3
>>                                 Space used (live): 1035147
>>                                 Space used (total): 3097633
>>                                 Number of Keys (estimate): 5760
>>                                 Memtable Columns Count: 8
>>                                 Memtable Data Size: 15376
>>                                 Memtable Switch Count: 7
>>                                 Read Count: 3971
>>                                 Read Latency: 0.023 ms.
>>                                 Write Count: 73
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 104
>>                                 Compacted row maximum size: 2299
>>                                 Compacted row mean size: 170
>>  
>>                                 Column Family: SavedHotelsInverted
>>                                 SSTable count: 1
>>                                 Space used (live): 1022963
>>                                 Space used (total): 4108255
>>                                 Number of Keys (estimate): 5504
>>                                 Memtable Columns Count: 8
>>                                 Memtable Data Size: 450
>>                                 Memtable Switch Count: 7
>>                                 Read Count: 3
>>                                 Read Latency: NaN ms.
>>                                 Write Count: 73
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 104
>>                                 Compacted row maximum size: 2299
>>                                 Compacted row mean size: 162
>>  
>>                                 Column Family: LastViewedHotelsInverted
>>                                 SSTable count: 4
>>                                 Space used (live): 243688384
>>                                 Space used (total): 362107541
>>                                 Number of Keys (estimate): 1257216
>>                                 Memtable Columns Count: 2917
>>                                 Memtable Data Size: 4254486
>>                                 Memtable Switch Count: 9
>>                                 Read Count: 2454
>>                                 Read Latency: 0.091 ms.
>>                                 Write Count: 18857
>>                                 Write Latency: 0.018 ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 73
>>                                 Compacted row maximum size: 51012
>>                                 Compacted row mean size: 170
>>  
>>                                 Column Family: SavedSearchRequest
>>                                 SSTable count: 7
>>                                 Space used (live): 7125532732
>>                                 Space used (total): 7266974627
>>                                 Number of Keys (estimate): 1526144
>>                                 Memtable Columns Count: 8854
>>                                 Memtable Data Size: 23875680
>>                                 Memtable Switch Count: 9
>>                                 Read Count: 4427
>>                                 Read Latency: 5.022 ms.
>>                                 Write Count: 54938
>>                                 Write Latency: 0.026 ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 925
>>                                 Compacted row maximum size: 4055269
>>                                 Compacted row mean size: 4912
>>  
>>                                 Column Family: HotelTariffs
>>                                 SSTable count: 1
>>                                 Space used (live): 46306159
>>                                 Space used (total): 46306159
>>                                 Number of Keys (estimate): 7680
>>                                 Memtable Columns Count: 0
>>                                 Memtable Data Size: 0
>>                                 Memtable Switch Count: 0
>>                                 Read Count: 0
>>                                 Read Latency: NaN ms.
>>                                 Write Count: 0
>>                                 Write Latency: NaN ms.
>>                                 Pending Tasks: 0
>>                                 Key cache: disabled
>>                                 Row cache: disabled
>>                                 Compacted row minimum size: 180
>>                                 Compacted row maximum size: 9887
>>                                 Compacted row mean size: 6535
>>  
>>  
>>  
>>  
>> Regards,
>> Shubham Srivastava | Technical Lead - Technology Development
>> +91 124 439 5428   |  MakeMyTrip.com, 103 Udyog Vihar Phase 1, Gurgaon, Haryana - 122 016, India
>> <image002.png>What's new? My Trip Rewards - An exclusive loyalty program for MakeMyTrip customers.
>> <image003.gif>
>> <image004.gif>
>> Office Map
>> <image005.gif>
>> Facebook
>> <image006.gif>
>> Twitter


Mime
View raw message