incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From elsif <elsif.t...@gmail.com>
Subject Re: cassandra 0.5.1 java.lang.OutOfMemoryError: Java heap space issue
Date Sat, 01 May 2010 00:03:40 GMT
I upgraded to 0.6.1 and was able to bring up all the nodes and make
queries. 

After adding some new data, the java vm ran out of memory on three of
the nodes. Cassandra continues to run for about 20 minutes before it
exits completely:

DEBUG [ROW-MUTATION-STAGE:2] 2010-04-30 16:02:27,298
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='5f1c96b3', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272668502,])])
applied.  Sending response to 749432@/10.1.29.20
DEBUG [ROW-MUTATION-STAGE:19] 2010-04-30 16:02:27,298
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='4dfdfc26', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272668513,])])
applied.  Sending response to 1744234@/10.1.29.18
DEBUG [ROW-MUTATION-STAGE:91] 2010-04-30 16:02:27,449
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='5efd19a7', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272668511,])])
applied.  Sending response to 969767@/10.1.29.22
ERROR [COMPACTION-POOL:1] 2010-04-30 16:02:27,295
DebuggableThreadPoolExecutor.java (line 94) Error in executor
futuretask                                                                               
                               
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
        at java.util.concurrent.FutureTask$Sync.innerGet(Unknown
Source)                                                     at
java.util.concurrent.FutureTask.get(Unknown
Source)                                                               at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.jav
a:86)                                                                                    
                                  
at
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
           
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)                                       
       at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.io.ByteArrayOutputStream.write(Unknown Source)
        at java.io.DataOutputStream.write(Unknown Source)
        at java.io.FilterOutputStream.write(Unknown Source)
        at
org.apache.cassandra.db.ColumnSerializer.writeName(ColumnSerializer.java:39)
        at
org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:61)
        at
org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:29)
        at
org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87)
        at
org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:99)
        at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:131)
        at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:41)
        at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
        at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
        at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
        at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:299)
        at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
        at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        ... 3 more
DEBUG [ROW-MUTATION-STAGE:64] 2010-04-30 16:02:27,247
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='298dff6c',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272668521,])])
DEBUG [ROW-MUTATION-STAGE:20] 2010-04-30 16:02:27,247
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='5ce38012',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272668520,])])
DEBUG [WRITE-/10.1.29.12] 2010-04-30 16:02:27,247
OutboundTcpConnection.java (line 142) attempting to connect to /10.1.29.12
.
.
.
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,235
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='29c4eeeb',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788,])])
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,235
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='29c4eeeb', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788,])])
applied.  Sending response to 756622@/10.1.29.20
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,236
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='5f25be67',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788,])])
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,236
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='5f25be67', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788,])])
applied.  Sending response to 756627@/10.1.29.20
DEBUG [ROW-MUTATION-STAGE:34] 2010-04-30 16:24:01,230
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='51f2f208',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272669785,])])
DEBUG [ROW-MUTATION-STAGE:98] 2010-04-30 16:24:01,230
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='29ed2f93', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272669778,])])
applied.  Sending response to 1017012@/10.1.29.16
DEBUG [ROW-MUTATION-STAGE:8] 2010-04-30 16:24:01,230
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='3d40ecfc',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669781,])])
DEBUG [ROW-MUTATION-STAGE:67] 2010-04-30 16:24:01,228
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='4d1e2910',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669780,])])



On 04/26/2010 06:48 PM, Jonathan Ellis wrote:
> 0.5 has a bug that allows it to OOM itself from replaying the log too
> fast.  You should upgrade to 0.6.1.
>
>   


Mime
View raw message