incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark <static.void....@gmail.com>
Subject Re: OOM exceptions
Date Sun, 06 Mar 2011 16:50:34 GMT
If its determined that this is due to a very large row, what are my options?

Thanks

On 3/5/11 7:11 PM, aaron morton wrote:
> First question is which version are you running ? Am guessing 0.6 
> something
>
> If you have OOM in the compaction thread it may be because of a very 
> large row. The CF information available through JConsole will give you 
> the max row size for the CF.
>
> Your setting for RowWarningThresholdInMB is 512, however the default 
> setting in 0.6.12 is 64mb. Here's the inline help from 
> storage-conf.xml for 0.6.12
>
> <!--
>    ~ Size of compacted row above which to log a warning.  If compacted
>    ~ rows do not fit in memory, Cassandra will crash.  (This is explained
>    ~ in http://wiki.apache.org/cassandra/CassandraLimitations and is
>    ~ scheduled to be fixed in 0.7.)  Large rows can also be a problem
>    ~ when row caching is enabled.
>   -->
> <RowWarningThresholdInMB>64</RowWarningThresholdInMB>
>
> In 0.7 the equivalent setting is in_memory_compaction_limit_in_mb , 
> here's the file help from the 0.7.3 conf file
>
> # Size limit for rows being compacted in memory.  Larger rows will spill
> # over to disk and use a slower two-pass compaction process.  A message
> # will be logged specifying the row key.
> in_memory_compaction_limit_in_mb: 64
> Hope that helps.
> Aaron
>
>
> On 5/03/2011, at 10:54 AM, Mark wrote:
>
>> Thats very nice of you. Thanks
>>
>> <Storage>
>> <ClusterName>MyCluster</ClusterName>
>> <AutoBootstrap>true</AutoBootstrap>
>> <HintedHandoffEnabled>true</HintedHandoffEnabled>
>> <IndexInterval>128</IndexInterval>
>>
>> <Keyspaces>
>> <Keyspace Name="MyCompany">
>> <ColumnFamily Name="SearchLog"
>>                     ColumnType="Super"
>>                     CompareWith="TimeUUIDType"
>>                     CompareSubcolumnsWith="BytesType"/>
>> <ColumnFamily Name="ProductLog"
>>                     ColumnType="Super"
>>                     CompareWith="TimeUUIDType"
>>                     CompareSubcolumnsWith="BytesType"/>
>> <ColumnFamily Name="RequestLog"
>>                     ColumnType="Super"
>>                     CompareWith="TimeUUIDType"
>>                     CompareSubcolumnsWith="BytesType"/>
>> <ColumnFamily Name="ClickLog"
>>                     ColumnType="Super"
>>                     CompareWith="TimeUUIDType"
>>                     CompareSubcolumnsWith="BytesType"/>
>> <ColumnFamily Name="ItemTranslation"
>>                     ColumnType="Super"
>>                     CompareWith="BytesType"
>>                     CompareSubcolumnsWith="BytesType"/>
>> <ColumnFamily Name="ItemTranslationIndex"
>>                     CompareWith="BytesType"/>
>> <ColumnFamily Name="RelatedObject"
>>                     CompareWith="LongType"/>
>>
>> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>> <ReplicationFactor>2</ReplicationFactor>
>> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>> </Keyspace>
>> </Keyspaces>
>>
>> <Authenticator>org.apache.cassandra.auth.AllowAllAuthenticator</Authenticator>
>> <Partitioner>org.apache.cassandra.dht.RandomPartitioner</Partitioner>
>> <InitialToken></InitialToken>
>>
>> <SavedCachesDirectory>/var/lib/cassandra/saved_caches</SavedCachesDirectory>
>> <CommitLogDirectory>/var/lib/cassandra/commitlog</CommitLogDirectory>
>> <DataFileDirectories>
>> <DataFileDirectory>/var/lib/cassandra/data</DataFileDirectory>
>> </DataFileDirectories>
>>
>> <Seeds>
>> <Seed>cassandra1</Seed>
>> <Seed>cassandra2</Seed>
>> </Seeds>
>>
>> <RpcTimeoutInMillis>10000</RpcTimeoutInMillis>
>> <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
>>
>> <ListenAddress></ListenAddress>
>> <StoragePort>7000</StoragePort>
>>
>> <ThriftAddress></ThriftAddress>
>> <ThriftPort>9160</ThriftPort>
>> <ThriftFramedTransport>false</ThriftFramedTransport>
>>
>> <DiskAccessMode>mmap_index_only</DiskAccessMode>
>>
>> <RowWarningThresholdInMB>512</RowWarningThresholdInMB>
>> <SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>
>> <FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
>> <FlushIndexBufferSizeInMB>8</FlushIndexBufferSizeInMB>
>> <ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
>> <MemtableThroughputInMB>64</MemtableThroughputInMB>
>> <BinaryMemtableThroughputInMB>256</BinaryMemtableThroughputInMB>
>> <MemtableFlushAfterMinutes>60</MemtableFlushAfterMinutes>
>>
>> <ConcurrentReads>8</ConcurrentReads>
>> <ConcurrentWrites>32</ConcurrentWrites>
>>
>> <CommitLogSync>periodic</CommitLogSync>
>> <CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>
>> <GCGraceSeconds>864000</GCGraceSeconds>
>>
>> <DoConsistencyChecksBoolean>true</DoConsistencyChecksBoolean>
>> </Storage>
>>
>>
>> On 3/4/11 1:05 PM, Narendra Sharma wrote:
>>> I have been through tuning for GC and OOM recently. If you can 
>>> provide the cassandra.yaml, I can help. Mostly I had to play with 
>>> memtable thresholds.
>>>
>>> Thanks,
>>> Naren
>>>
>>> On Fri, Mar 4, 2011 at 12:43 PM, Mark <static.void.dev@gmail.com 
>>> <mailto:static.void.dev@gmail.com>> wrote:
>>>
>>>     We have 7 column families and we are not using the default key
>>>     cache (200000).
>>>
>>>     These were our initial settings so it was not in response to
>>>     anything. Would you recommend anything else? Thanks
>>>
>>>
>>>
>>>     On 3/4/11 12:34 PM, Chris Burroughs wrote:
>>>
>>>         - Are you using a key cache?  How many keys do you have?
>>>          Across how
>>>         many column families
>>>
>>>         You configuration is unusual both in terms of not setting
>>>         min heap ==
>>>         max heap and the percentage of available RAM used for the
>>>         heap.  Did you
>>>         change the heap size in response to errors or for another
>>>         reason?
>>>
>>>         On 03/04/2011 03:25 PM, Mark wrote:
>>>
>>>             This happens during compaction and we are not using the
>>>             RowsCached
>>>             attribute.
>>>
>>>             Our initial/max heap are 2 and 6 respectively and we
>>>             have 8 gigs in
>>>             these machines.
>>>
>>>             Thanks
>>>
>>>             On 3/4/11 12:05 PM, Chris Burroughs wrote:
>>>
>>>                 - Does this occur only during compaction or at
>>>                 seemingly random times?
>>>                 - How large is your heap?  What jvm settings are you
>>>                 using? How much
>>>                 physical RAM do you have?
>>>                 - Do you have the row and/or key cache enabled?  How
>>>                 are they
>>>                 configured?  How large are they when the OOM is thrown?
>>>
>>>                 On 03/04/2011 02:38 PM, Mark Miller wrote:
>>>
>>>                     Other than adding more memory to the machine is
>>>                     there a way to solve
>>>                     this? Please help. Thanks
>>>
>>>                     ERROR [COMPACTION-POOL:1] 2011-03-04
>>>                     11:11:44,891 CassandraDaemon.java
>>>                     (line
>>>                     org.apache.cassandra.thrift.CassandraDaemon$1)
>>>                     Uncaught exception
>>>                     in thread Thread[COMPACTION-POOL:1,5,main]
>>>                     java.lang.OutOfMemoryError: Java heap space
>>>                          at java.util.Arrays.copyOf(Arrays.java:2798)
>>>                          at
>>>                     java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:111)
>>>                          at
>>>                     java.io.DataOutputStream.write(DataOutputStream.java:107)
>>>                          at
>>>                     java.io.FilterOutputStream.write(FilterOutputStream.java:97)
>>>                          at
>>>                     org.apache.cassandra.utils.FBUtilities.writeByteArray(FBUtilities.java:298)
>>>
>>>                          at
>>>                     org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:66)
>>>
>>>
>>>                          at
>>>                     org.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:311)
>>>
>>>
>>>                          at
>>>                     org.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:284)
>>>
>>>
>>>                          at
>>>                     org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87)
>>>
>>>
>>>                          at
>>>                     org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:99)
>>>
>>>
>>>                          at
>>>                     org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:140)
>>>
>>>
>>>                          at
>>>                     org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
>>>
>>>
>>>                          at
>>>                     org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
>>>
>>>
>>>                          at
>>>                     com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
>>>
>>>
>>>                          at
>>>                     com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
>>>
>>>
>>>                          at
>>>                     org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>>>
>>>
>>>                          at
>>>                     org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>>>
>>>
>>>                          at
>>>                     org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:294)
>>>
>>>
>>>                          at
>>>                     org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:101)
>>>
>>>
>>>                          at
>>>                     org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:82)
>>>
>>>                          at
>>>                     java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>                          at
>>>                     java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>                          at
>>>                     java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>
>>>
>>>                          at
>>>                     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>
>>>
>>>                          at java.lang.Thread.run(Thread.java:636)
>>>
>>>
>

Mime
View raw message