Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.212.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAQKod0pd3wSBAPCRhbps1p=B6s8KUfSPkaKBT5f66CWBD8MLg@mail.gmail.com>
References: 
 <CAAQKod0pd3wSBAPCRhbps1p=B6s8KUfSPkaKBT5f66CWBD8MLg@mail.gmail.com>
Date: Thu, 27 Feb 2014 16:38:35 -0500
Message-ID: 
 <CAENxBwzGhunt9cdDD_-z3yj1Q+VPiBT6xQ80B2JYYgRsaaEZKg@mail.gmail.com>
Subject: Re: OOM while performing major compaction
From: Edward Capriolo <edlinuxguru@gmail.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Content-Type: multipart/alternative; boundary=f46d043c815eb9d38504f36a247d

--f46d043c815eb9d38504f36a247d
Content-Type: text/plain; charset=ISO-8859-1

One big downside about major compaction is that (depending on your
cassandra version) the bloom filters size is pre-calculated. Thus cassandra
needs enough heap for your existing 33 k+ sstables and the new large
compacted one. In the past this happened to us when the compaction thread
got hung up, and the sstables grew. When this happens I delete the data
directory and jump in a fresh node. The time to recover from that sstable
build up can be huge even with multi-threaded compaction.


On Thu, Feb 27, 2014 at 2:09 PM, Nish garg <pipelines@gmail.com> wrote:

> I am having OOM during major compaction on one of the column family where
> there are lot of SStables (33000) to be compacted. Is there any other way
> for them to be compacted? Any help will be really appreciated.
>
> Here are the details
>
>  /opt/cassandra/current/bin/nodetool -h us1emscsm-01  compact tomcat
> sessions
> Error occurred during compaction
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
>  at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>  at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>  at
> org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:334)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1691)
>  at
> org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2168)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>  at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>  at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>  at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>  at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>  at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>  at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>  at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>  at sun.rmi.transport.Transport$1.run(Transport.java:159)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>  at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>  at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>  at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.OutOfMemoryError: Java heap space
>  at
> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:77)
>  at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
>  at
> org.apache.cassandra.io.compress.CompressedThrottledReader.<init>(CompressedThrottledReader.java:38)
>  at
> org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1212)
>  at
> org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:54)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1032)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1044)
>  at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:157)
>  at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:163)
>  at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:117)
>  at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>  at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>  at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
>  at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
>  at
> org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:355)
>  at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>  ... 3 more
>
>
> Also
>
>
> /opt/cassandra/current/bin/nodetool -h us1emscsm-01
> getcompactionthreshold tomcat sessions
> Current compaction thresholds for tomcat/sessions:
>  min = 4,  max = 32
>
>
>  /opt/cassandra/current/bin/nodetool -h us1emscsm-01
> getcompactionthroughput tomcat sessions
> Current compaction throughput: 16 MB/s
>
>
>
> /opt/cassandra/current/bin/nodetool -h us1emscsm-01  compactionstats
> pending tasks: 167
> Active compaction remaining time :        n/a
>
>
>

--f46d043c815eb9d38504f36a247d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">One big downside about major compaction is that (depending=
 on your cassandra version) the bloom filters size is pre-calculated. Thus =
cassandra needs enough heap for your existing 33 k+ sstables and the new la=
rge compacted one. In the past this happened to us when the compaction thre=
ad got hung up, and the sstables grew. When this happens I delete the data =
directory and jump in a fresh node. The time to recover from that sstable b=
uild up can be huge even with multi-threaded compaction.<br>

<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Thu, Feb 2=
7, 2014 at 2:09 PM, Nish garg <span dir=3D"ltr">&lt;<a href=3D"mailto:pipel=
ines@gmail.com" target=3D"_blank">pipelines@gmail.com</a>&gt;</span> wrote:=
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>I am having OOM during=
 major compaction on one of the column family where there are lot of SStabl=
es (33000)=A0to be compacted. Is there any other way for them to be compact=
ed? Any help will be really appreciated.<br>


<br>Here are the details<br><br>=A0/opt/cassandra/current/bin/nodetool -h u=
s1emscsm-01=A0 compact tomcat sessions<br>Error occurred during compaction<=
br>java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Jav=
a heap space<br>


=A0at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)<br=
>=A0at java.util.concurrent.FutureTask.get(FutureTask.java:83)<br>=A0at org=
.apache.cassandra.db.compaction.CompactionManager.performMaximal(Compaction=
Manager.java:334)<br>


=A0at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(Column=
FamilyStore.java:1691)<br>=A0at org.apache.cassandra.service.StorageService=
.forceTableCompaction(StorageService.java:2168)<br>=A0at sun.reflect.Native=
MethodAccessorImpl.invoke0(Native Method)<br>


=A0at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.=
java:39)<br>=A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin=
gMethodAccessorImpl.java:25)<br>=A0at java.lang.reflect.Method.invoke(Metho=
d.java:597)<br>


=A0at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMB=
eanIntrospector.java:93)<br>=A0at com.sun.jmx.mbeanserver.StandardMBeanIntr=
ospector.invokeM2(StandardMBeanIntrospector.java:27)<br>=A0at com.sun.jmx.m=
beanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)<br>


=A0at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)<br=
>=A0at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)<b=
r>=A0at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Defaul=
tMBeanServerInterceptor.java:836)<br>


=A0at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761=
)<br>=A0at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMICon=
nectionImpl.java:1427)<br>=A0at javax.management.remote.rmi.RMIConnectionIm=
pl.access$200(RMIConnectionImpl.java:72)<br>


=A0at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run=
(RMIConnectionImpl.java:1265)<br>=A0at javax.management.remote.rmi.RMIConne=
ctionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)<br>=A0at javax=
.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)=
<br>


=A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)<br>=A0at =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3=
9)<br>=A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMetho=
dAccessorImpl.java:25)<br>


=A0at java.lang.reflect.Method.invoke(Method.java:597)<br>=A0at sun.rmi.ser=
ver.UnicastServerRef.dispatch(UnicastServerRef.java:305)<br>=A0at sun.rmi.t=
ransport.Transport$1.run(Transport.java:159)<br>=A0at java.security.AccessC=
ontroller.doPrivileged(Native Method)<br>


=A0at sun.rmi.transport.Transport.serviceCall(Transport.java:155)<br>=A0at =
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)<br=
>=A0at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTranspo=
rt.java:790)<br>


=A0at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport=
.java:649)<br>=A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(=
ThreadPoolExecutor.java:886)<br>=A0at java.util.concurrent.ThreadPoolExecut=
or$Worker.run(ThreadPoolExecutor.java:908)<br>


=A0at java.lang.Thread.run(Thread.java:662)<br>Caused by: java.lang.OutOfMe=
moryError: Java heap space<br>=A0at org.apache.cassandra.io.util.RandomAcce=
ssReader.&lt;init&gt;(RandomAccessReader.java:77)<br>=A0at org.apache.cassa=
ndra.io.compress.CompressedRandomAccessReader.&lt;init&gt;(CompressedRandom=
AccessReader.java:75)<br>


=A0at org.apache.cassandra.io.compress.CompressedThrottledReader.&lt;init&g=
t;(CompressedThrottledReader.java:38)<br>=A0at org.apache.cassandra.io.comp=
ress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)<br>

=A0at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableR=
eader.java:1212)<br>
=A0at org.apache.cassandra.io.sstable.SSTableScanner.&lt;init&gt;(SSTableSc=
anner.java:54)<br>=A0at org.apache.cassandra.io.sstable.SSTableReader.getDi=
rectScanner(SSTableReader.java:1032)<br>=A0at org.apache.cassandra.io.sstab=
le.SSTableReader.getDirectScanner(SSTableReader.java:1044)<br>


=A0at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScan=
ners(AbstractCompactionStrategy.java:157)<br>=A0at org.apache.cassandra.db.=
compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrateg=
y.java:163)<br>


=A0at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionT=
ask.java:117)<br>=A0at org.apache.cassandra.io.util.DiskAwareRunnable.runMa=
yThrow(DiskAwareRunnable.java:48)<br>=A0at org.apache.cassandra.utils.Wrapp=
edRunnable.run(WrappedRunnable.java:28)<br>


=A0at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(Com=
pactionTask.java:58)<br>=A0at org.apache.cassandra.db.compaction.AbstractCo=
mpactionTask.execute(AbstractCompactionTask.java:60)<br>=A0at org.apache.ca=
ssandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.jav=
a:355)<br>


=A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:2=
8)<br>=A0at java.util.concurrent.Executors$RunnableAdapter.call(Executors.j=
ava:441)<br>=A0at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.=
java:303)<br>


=A0at java.util.concurrent.FutureTask.run(FutureTask.java:138)<br>=A0... 3 =
more<br><br><br>Also <br><br><br>/opt/cassandra/current/bin/nodetool -h us1=
emscsm-01=A0 getcompactionthreshold tomcat sessions<br>Current compaction t=
hresholds for tomcat/sessions: <br>


=A0min =3D 4,=A0 max =3D 32</div><div>=A0</div><div><br>=A0/opt/cassandra/c=
urrent/bin/nodetool -h us1emscsm-01=A0 getcompactionthroughput tomcat sessi=
ons<br>Current compaction throughput: 16 MB/s<br><br><br><br>/opt/cassandra=
/current/bin/nodetool -h us1emscsm-01=A0 compactionstats<br>


pending tasks: 167<br>Active compaction remaining time :=A0=A0=A0=A0=A0=A0=
=A0 n/a<br><br><br></div></div>
</blockquote></div><br></div></div>

--f46d043c815eb9d38504f36a247d--