incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nish garg <pipeli...@gmail.com>
Subject OOM while performing major compaction
Date Thu, 27 Feb 2014 19:09:18 GMT
I am having OOM during major compaction on one of the column family where
there are lot of SStables (33000) to be compacted. Is there any other way
for them to be compacted? Any help will be really appreciated.

Here are the details

 /opt/cassandra/current/bin/nodetool -h us1emscsm-01  compact tomcat
sessions
Error occurred during compaction
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
heap space
 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
 at java.util.concurrent.FutureTask.get(FutureTask.java:83)
 at
org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:334)
 at
org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1691)
 at
org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2168)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
 at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
 at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
 at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
 at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
 at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
 at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
 at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
 at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
 at sun.rmi.transport.Transport$1.run(Transport.java:159)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
 at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
 at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
 at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:77)
 at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
 at
org.apache.cassandra.io.compress.CompressedThrottledReader.<init>(CompressedThrottledReader.java:38)
 at
org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
 at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1212)
 at
org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:54)
 at
org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1032)
 at
org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1044)
 at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:157)
 at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:163)
 at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:117)
 at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
 at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
 at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
 at
org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:355)
 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 ... 3 more


Also


/opt/cassandra/current/bin/nodetool -h us1emscsm-01  getcompactionthreshold
tomcat sessions
Current compaction thresholds for tomcat/sessions:
 min = 4,  max = 32


 /opt/cassandra/current/bin/nodetool -h us1emscsm-01
getcompactionthroughput tomcat sessions
Current compaction throughput: 16 MB/s



/opt/cassandra/current/bin/nodetool -h us1emscsm-01  compactionstats
pending tasks: 167
Active compaction remaining time :        n/a

Mime
View raw message