Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9C0411022D for ; Thu, 27 Feb 2014 19:09:48 +0000 (UTC) Received: (qmail 60065 invoked by uid 500); 27 Feb 2014 19:09:45 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 60022 invoked by uid 500); 27 Feb 2014 19:09:45 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 60014 invoked by uid 99); 27 Feb 2014 19:09:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Feb 2014 19:09:44 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of pipelines@gmail.com designates 209.85.215.194 as permitted sender) Received: from [209.85.215.194] (HELO mail-ea0-f194.google.com) (209.85.215.194) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Feb 2014 19:09:39 +0000 Received: by mail-ea0-f194.google.com with SMTP id h10so1109244eak.5 for ; Thu, 27 Feb 2014 11:09:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=fsvQQ4IiZl1GnRb+DsMwVyyJwZylevGC6/a6UDHzDzk=; b=LQfKGzzn7V+e1LJ+U70+pKhYYe5uYhLVVSewB+Q1EBvDYCjH4UK1rDz6Gc7Aqh+jQo idGyQ741WSv2iU1r01QvU0O6Fe67DYNcDlBMvit2Q4rBl4t+RX8ic8aGgsA7kGOcMiEq 1n0wGEFoSOmii+QwXbhHjKg4JUymtCDKSvsRbP4xZuq92FmrebhBE8wavpbzk/qpPm7j hKqPbvvQKPIaIYuj4fdOv9bx1n4XtZKeVjHMwKm1FHfzA/xylMZgbu2PV4oSxeg8iKfx OEPrh9p1nzT5/pdau/bs3J1yLcRKX59M6dTeVLm+KYwcHdErlGCnTO48kOY2f/nOuBrh mtug== MIME-Version: 1.0 X-Received: by 10.204.103.131 with SMTP id k3mr59171bko.81.1393528158221; Thu, 27 Feb 2014 11:09:18 -0800 (PST) Received: by 10.205.98.7 with HTTP; Thu, 27 Feb 2014 11:09:18 -0800 (PST) Date: Thu, 27 Feb 2014 13:09:18 -0600 Message-ID: Subject: OOM while performing major compaction From: Nish garg To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11334a16db1d6604f3680e71 X-Virus-Checked: Checked by ClamAV on apache.org --001a11334a16db1d6604f3680e71 Content-Type: text/plain; charset=ISO-8859-1 I am having OOM during major compaction on one of the column family where there are lot of SStables (33000) to be compacted. Is there any other way for them to be compacted? Any help will be really appreciated. Here are the details /opt/cassandra/current/bin/nodetool -h us1emscsm-01 compact tomcat sessions Error occurred during compaction java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:334) at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1691) at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2168) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:77) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.(CompressedRandomAccessReader.java:75) at org.apache.cassandra.io.compress.CompressedThrottledReader.(CompressedThrottledReader.java:38) at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1212) at org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:54) at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1032) at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1044) at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:157) at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:163) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:117) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:355) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more Also /opt/cassandra/current/bin/nodetool -h us1emscsm-01 getcompactionthreshold tomcat sessions Current compaction thresholds for tomcat/sessions: min = 4, max = 32 /opt/cassandra/current/bin/nodetool -h us1emscsm-01 getcompactionthroughput tomcat sessions Current compaction throughput: 16 MB/s /opt/cassandra/current/bin/nodetool -h us1emscsm-01 compactionstats pending tasks: 167 Active compaction remaining time : n/a --001a11334a16db1d6604f3680e71 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I am having OOM during major compaction on one of the= column family where there are lot of SStables (33000)=A0to be compacted. I= s there any other way for them to be compacted? Any help will be really app= reciated.

Here are the details

=A0/opt/cassandra/current/bin/nodetool -h u= s1emscsm-01=A0 compact tomcat sessions
Error occurred during compaction<= br>java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Jav= a heap space
=A0at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)=A0at java.util.concurrent.FutureTask.get(FutureTask.java:83)
=A0at org= .apache.cassandra.db.compaction.CompactionManager.performMaximal(Compaction= Manager.java:334)
=A0at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(Column= FamilyStore.java:1691)
=A0at org.apache.cassandra.service.StorageService= .forceTableCompaction(StorageService.java:2168)
=A0at sun.reflect.Native= MethodAccessorImpl.invoke0(Native Method)
=A0at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.= java:39)
=A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin= gMethodAccessorImpl.java:25)
=A0at java.lang.reflect.Method.invoke(Metho= d.java:597)
=A0at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMB= eanIntrospector.java:93)
=A0at com.sun.jmx.mbeanserver.StandardMBeanIntr= ospector.invokeM2(StandardMBeanIntrospector.java:27)
=A0at com.sun.jmx.m= beanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
=A0at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)=A0at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)=A0at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Defaul= tMBeanServerInterceptor.java:836)
=A0at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761= )
=A0at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMICon= nectionImpl.java:1427)
=A0at javax.management.remote.rmi.RMIConnectionIm= pl.access$200(RMIConnectionImpl.java:72)
=A0at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run= (RMIConnectionImpl.java:1265)
=A0at javax.management.remote.rmi.RMIConne= ctionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
=A0at javax= .management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)=
=A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
=A0at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3= 9)
=A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMetho= dAccessorImpl.java:25)
=A0at java.lang.reflect.Method.invoke(Method.java:597)
=A0at sun.rmi.ser= ver.UnicastServerRef.dispatch(UnicastServerRef.java:305)
=A0at sun.rmi.t= ransport.Transport$1.run(Transport.java:159)
=A0at java.security.AccessC= ontroller.doPrivileged(Native Method)
=A0at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
=A0at = sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)=A0at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTranspo= rt.java:790)
=A0at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport= .java:649)
=A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(= ThreadPoolExecutor.java:886)
=A0at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:908)
=A0at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMe= moryError: Java heap space
=A0at org.apache.cassandra.io.util.RandomAcce= ssReader.<init>(RandomAccessReader.java:77)
=A0at org.apache.cassa= ndra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandom= AccessReader.java:75)
=A0at org.apache.cassandra.io.compress.CompressedThrottledReader.<init&g= t;(CompressedThrottledReader.java:38)
=A0at org.apache.cassandra.io.comp= ress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
= =A0at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableR= eader.java:1212)
=A0at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableSc= anner.java:54)
=A0at org.apache.cassandra.io.sstable.SSTableReader.getDi= rectScanner(SSTableReader.java:1032)
=A0at org.apache.cassandra.io.sstab= le.SSTableReader.getDirectScanner(SSTableReader.java:1044)
=A0at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScan= ners(AbstractCompactionStrategy.java:157)
=A0at org.apache.cassandra.db.= compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrateg= y.java:163)
=A0at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionT= ask.java:117)
=A0at org.apache.cassandra.io.util.DiskAwareRunnable.runMa= yThrow(DiskAwareRunnable.java:48)
=A0at org.apache.cassandra.utils.Wrapp= edRunnable.run(WrappedRunnable.java:28)
=A0at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(Com= pactionTask.java:58)
=A0at org.apache.cassandra.db.compaction.AbstractCo= mpactionTask.execute(AbstractCompactionTask.java:60)
=A0at org.apache.ca= ssandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.jav= a:355)
=A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:2= 8)
=A0at java.util.concurrent.Executors$RunnableAdapter.call(Executors.j= ava:441)
=A0at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.= java:303)
=A0at java.util.concurrent.FutureTask.run(FutureTask.java:138)
=A0... 3 = more


Also


/opt/cassandra/current/bin/nodetool -h us1= emscsm-01=A0 getcompactionthreshold tomcat sessions
Current compaction t= hresholds for tomcat/sessions:
=A0min =3D 4,=A0 max =3D 32
=A0

=A0/opt/cassandra/c= urrent/bin/nodetool -h us1emscsm-01=A0 getcompactionthroughput tomcat sessi= ons
Current compaction throughput: 16 MB/s



/opt/cassandra= /current/bin/nodetool -h us1emscsm-01=A0 compactionstats
pending tasks: 167
Active compaction remaining time :=A0=A0=A0=A0=A0=A0= =A0 n/a


--001a11334a16db1d6604f3680e71--