cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: cassandra 0.5.1 java.lang.OutOfMemoryError: Java heap space issue
Date Tue, 27 Apr 2010 01:48:25 GMT
0.5 has a bug that allows it to OOM itself from replaying the log too
fast.  You should upgrade to 0.6.1.

On Mon, Apr 26, 2010 at 12:11 PM, elsif <elsif.then@gmail.com> wrote:
>
> Hello.  I have a six node cassandra cluster running on modest hardware
> with 1G of heap assigned to cassandra.  After inserting about 245
> million rows of data, cassandra failed with a
> java.lang.OutOfMemoryError: Java heap space error.  I rasied the java
> heap to 2G, but still get the same error when trying to restart cassandra.
>
> I am using Cassandra 0.5.1 with Sun jre1.6.0_18.
>
> Any thoughts on how to resolve this issue are greatly appreciated.
>
> Here are log excerpts from two of the nodes:
>
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
> SliceQueryFilter.java (line 116) collecting SuperColumn(dcf9f19e
> [0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
> SliceQueryFilter.java (line 116) collecting SuperColumn(dd04bf9c
> [0a011d0c,0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
> SliceQueryFilter.java (line 116) collecting SuperColumn(dd08981a
> [0a011d0c,0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
> SliceQueryFilter.java (line 116) collecting SuperColumn(dd7f7ac9
> [0a011d0c,0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
> SliceQueryFilter.java (line 116) collecting SuperColumn(dde1d4cf
> [0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
> SliceQueryFilter.java (line 116) collecting SuperColumn(de32aec3
> [0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
> SliceQueryFilter.java (line 116) collecting SuperColumn(de378105
> [0a011d0c,0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
> SliceQueryFilter.java (line 116) collecting SuperColumn(deb5d591
> [0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
> SliceQueryFilter.java (line 116) collecting SuperColumn(ded75dee
> [0a011d0c,0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
> SliceQueryFilter.java (line 116) collecting SuperColumn(defe3445
> [0a011d0c,0a011d0d,])
>  INFO [FLUSH-TIMER] 2010-04-23 16:20:00,071 ColumnFamilyStore.java (line
> 393) IpTag has reached its threshold; switching in a fresh Memtable
>  INFO [FLUSH-TIMER] 2010-04-23 16:20:00,072 ColumnFamilyStore.java (line
> 1035) Enqueuing flush of Memtable(IpTag)@7816
>  INFO [FLUSH-SORTER-POOL:1] 2010-04-23 16:20:00,072 Memtable.java (line
> 183) Sorting Memtable(IpTag)@7816
>  INFO [FLUSH-WRITER-POOL:1] 2010-04-23 16:20:00,107 Memtable.java (line
> 192) Writing Memtable(IpTag)@7816
> DEBUG [Timer-0] 2010-04-23 16:20:00,130 LoadDisseminator.java (line 39)
> Disseminating load info ...
> ERROR [ROW-MUTATION-STAGE:41] 2010-04-23 16:20:00,348
> CassandraDaemon.java (line 71) Fatal exception in thread
> Thread[ROW-MUTATION-STAGE:41,5,main]
> java.lang.OutOfMemoryError: Java heap space
>        at java.util.Arrays.copyOfRange(Unknown Source)
>        at java.lang.String.<init>(Unknown Source)
>        at java.lang.StringBuilder.toString(Unknown Source)
>        at
> org.apache.cassandra.db.marshal.AbstractType.getColumnsString(AbstractType.java:87)
>        at
> org.apache.cassandra.db.ColumnFamily.toString(ColumnFamily.java:344)
>        at
> org.apache.commons.lang.ObjectUtils.toString(ObjectUtils.java:241)
>        at org.apache.commons.lang.StringUtils.join(StringUtils.java:3073)
>        at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133)
>        at
> org.apache.cassandra.db.RowMutation.toString(RowMutation.java:263)
>        at java.lang.String.valueOf(Unknown Source)
>        at java.lang.StringBuilder.append(Unknown Source)
>        at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46)
>        at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>        at java.lang.Thread.run(Unknown Source)
>
> ---
>
> DEBUG [main] 2010-04-23 17:15:45,501 CommitLog.java (line 312) Reading
> mutation at 57527476
> DEBUG [main] 2010-04-23 17:16:11,375 CommitLog.java (line 340) replaying
> mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c5c0,])}
> DEBUG [main] 2010-04-23 17:16:45,293 CommitLog.java (line 312) Reading
> mutation at 57527686
> DEBUG [main] 2010-04-23 17:16:45,294 CommitLog.java (line 340) replaying
> mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c5fb,])}
> DEBUG [main] 2010-04-23 17:16:54,311 CommitLog.java (line 312) Reading
> mutation at 57527919
> DEBUG [main] 2010-04-23 17:17:46,344 CommitLog.java (line 340) replaying
> mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c5fb,])}
> DEBUG [main] 2010-04-23 17:17:55,530 CommitLog.java (line 312) Reading
> mutation at 57528129
> DEBUG [main] 2010-04-23 17:18:20,266 CommitLog.java (line 340) replaying
> mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c607,])}
> DEBUG [main] 2010-04-23 17:18:38,273 CommitLog.java (line 312) Reading
> mutation at 57528362
> DEBUG [main] 2010-04-23 17:21:53,966 CommitLog.java (line 340) replaying
> mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c607,])}
> DEBUG [main] 2010-04-23 17:24:48,032 CommitLog.java (line 312) Reading
> mutation at 57528572
> ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,932
> CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
> Connection(idle),5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,952
> CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
> Connection(idle),5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,952
> CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
> Connection(idle),5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
>        at java.io.BufferedInputStream.<init>(Unknown Source)
>        at java.io.BufferedInputStream.<init>(Unknown Source)
>        at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
>        at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>        at java.lang.Thread.run(Unknown Source)
> ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,966
> CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
> Connection(idle),5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [main] 2010-04-23 17:36:38,966 CassandraDaemon.java (line 184)
> Exception encountered during startup.
> java.lang.OutOfMemoryError: Java heap space
> ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,981
> CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
> Connection(idle),5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
>
>
> Here is my current configuration:
>
> <Partitioner>org.apache.cassandra.dht.OrderPreservingPartitioner</Partitioner>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
> <ReplicationFactor>3</ReplicationFactor>
> <RpcTimeoutInMillis>30000</RpcTimeoutInMillis>
>  <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
>  <SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>
>  <FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
>  <ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
>  <MemtableSizeInMB>64</MemtableSizeInMB>
>  <MemtableObjectCountInMillions>0.1</MemtableObjectCountInMillions>
>  <MemtableFlushAfterMinutes>60</MemtableFlushAfterMinutes>
>  <ConcurrentReads>8</ConcurrentReads>
>  <ConcurrentWrites>100</ConcurrentWrites>
>  <CommitLogSync>periodic</CommitLogSync>
>  <CommitLogSyncPeriodInMS>1000</CommitLogSyncPeriodInMS>
>  <GCGraceSeconds>864000</GCGraceSeconds>
>  <BinaryMemtableSizeInMB>256</BinaryMemtableSizeInMB>
>
> Ring status:
>
> Address       Status     Load         Range
>         Ring
>                                       f
> 10.1.29.12    Down       7.26 GB      0
>         |<--|
> 10.1.29.13    Up         3.97 GB      3
>         |   ^
> 10.1.29.14    Up         7.73 GB      6
>         v   |
> 10.1.29.15    Down       14.27 GB     9
>         |   ^
> 10.1.29.16    Up         15.42 GB     c
>         v   |
> 10.1.29.17    Down       12.67 GB     f
>         |-->|
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message