incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: OOM while performing major compaction
Date Thu, 27 Feb 2014 21:38:35 GMT
One big downside about major compaction is that (depending on your
cassandra version) the bloom filters size is pre-calculated. Thus cassandra
needs enough heap for your existing 33 k+ sstables and the new large
compacted one. In the past this happened to us when the compaction thread
got hung up, and the sstables grew. When this happens I delete the data
directory and jump in a fresh node. The time to recover from that sstable
build up can be huge even with multi-threaded compaction.


On Thu, Feb 27, 2014 at 2:09 PM, Nish garg <pipelines@gmail.com> wrote:

> I am having OOM during major compaction on one of the column family where
> there are lot of SStables (33000) to be compacted. Is there any other way
> for them to be compacted? Any help will be really appreciated.
>
> Here are the details
>
>  /opt/cassandra/current/bin/nodetool -h us1emscsm-01  compact tomcat
> sessions
> Error occurred during compaction
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
>  at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>  at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>  at
> org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:334)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1691)
>  at
> org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2168)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>  at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>  at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>  at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>  at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>  at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>  at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>  at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>  at sun.rmi.transport.Transport$1.run(Transport.java:159)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>  at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>  at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>  at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.OutOfMemoryError: Java heap space
>  at
> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:77)
>  at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
>  at
> org.apache.cassandra.io.compress.CompressedThrottledReader.<init>(CompressedThrottledReader.java:38)
>  at
> org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1212)
>  at
> org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:54)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1032)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1044)
>  at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:157)
>  at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:163)
>  at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:117)
>  at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>  at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>  at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
>  at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
>  at
> org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:355)
>  at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>  ... 3 more
>
>
> Also
>
>
> /opt/cassandra/current/bin/nodetool -h us1emscsm-01
> getcompactionthreshold tomcat sessions
> Current compaction thresholds for tomcat/sessions:
>  min = 4,  max = 32
>
>
>  /opt/cassandra/current/bin/nodetool -h us1emscsm-01
> getcompactionthroughput tomcat sessions
> Current compaction throughput: 16 MB/s
>
>
>
> /opt/cassandra/current/bin/nodetool -h us1emscsm-01  compactionstats
> pending tasks: 167
> Active compaction remaining time :        n/a
>
>
>

Mime
View raw message