Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 74.125.83.175 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=fkMxh4wURMxPyhwStEcZcB1Q8YmYuOVv6MJBi4+3j81c4vVju3E3f0WpukXQRcnIbW
         5r/KyBwLDwtZPWJsS7GoEEOQxngz1B+l+llFrPuGqi1fw7JoK+/K37ebzCHhCRMmgUlS
         k/E0jgUE2vZE7R5cshmnztrsEDmAZUn3T4nQo=
MIME-Version: 1.0
In-Reply-To: <1297112561016-6001773.post@n2.nabble.com>
References: <1297112561016-6001773.post@n2.nabble.com>
Date: Mon, 7 Feb 2011 17:24:39 -0600
Message-ID: <AANLkTikvzL4=-p9dt1scTFL5Y4JQ8bvgJe7WqGQA7kAY@mail.gmail.com>
Subject: Re: Java bombs during compaction, please help
From: Jonathan Ellis <jbellis@gmail.com>
To: user <user@cassandra.apache.org>
Cc: cassandra-user <cassandra-user@incubator.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I've patched ColumnSortedMap on the 0.7 branch to not swallow the
IOException it's getting.

On Mon, Feb 7, 2011 at 3:02 PM, buddhasystem <potekhin@bnl.gov> wrote:
>
> Hello,
> one node in my 3-machine cluster cannot perform compaction. I tried multi=
ple
> times, it ran out of heap space once and I increased it. Now I'm getting =
the
> dump below (after it does run for a few minutes). I hope somebody can she=
d a
> little light on what' going on, because I'm at a loss and this is a real
> show stopper.
>
>
> [me@mymachine]~/cassandra-test% Error occured while compacting keyspace
> Tracer
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerGet(Unknown S=
ource)
> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.get(Unknown Source)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.=
java:186)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFami=
lyStore.java:1766)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.service.StorageService.forceTableCompaction(StorageS=
ervice.java:1236)
> =A0 =A0 =A0 =A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Met=
hod)
> =A0 =A0 =A0 =A0at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Sou=
rce)
> =A0 =A0 =A0 =A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown=
 Source)
> =A0 =A0 =A0 =A0at java.lang.reflect.Method.invoke(Unknown Source)
> =A0 =A0 =A0 =A0at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source=
)
> =A0 =A0 =A0 =A0at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source=
)
> =A0 =A0 =A0 =A0at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unkno=
wn Source)
> =A0 =A0 =A0 =A0at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Sou=
rce)
> =A0 =A0 =A0 =A0at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Sou=
rce)
> =A0 =A0 =A0 =A0at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Sour=
ce)
> =A0 =A0 =A0 =A0at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown S=
ource)
> =A0 =A0 =A0 =A0at javax.management.remote.rmi.RMIConnectionImpl.doOperati=
on(Unknown
> Source)
> =A0 =A0 =A0 =A0at javax.management.remote.rmi.RMIConnectionImpl.access$20=
0(Unknown
> Source)
> =A0 =A0 =A0 =A0at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unk=
nown
> Source)
> =A0 =A0 =A0 =A0at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unkno=
wn
> Source)
> =A0 =A0 =A0 =A0at javax.management.remote.rmi.RMIConnectionImpl.invoke(Un=
known
> Source)
> =A0 =A0 =A0 =A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Met=
hod)
> =A0 =A0 =A0 =A0at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Sou=
rce)
> =A0 =A0 =A0 =A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown=
 Source)
> =A0 =A0 =A0 =A0at java.lang.reflect.Method.invoke(Unknown Source)
> =A0 =A0 =A0 =A0at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source=
)
> =A0 =A0 =A0 =A0at sun.rmi.transport.Transport$1.run(Unknown Source)
> =A0 =A0 =A0 =A0at java.security.AccessController.doPrivileged(Native Meth=
od)
> =A0 =A0 =A0 =A0at sun.rmi.transport.Transport.serviceCall(Unknown Source)
> =A0 =A0 =A0 =A0at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unkno=
wn Source)
> =A0 =A0 =A0 =A0at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.ru=
n0(Unknown
> Source)
> =A0 =A0 =A0 =A0at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.ru=
n(Unknown
> Source)
> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(=
Unknown
> Source)
> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unkn=
own
> Source)
> =A0 =A0 =A0 =A0at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.NullPointerException
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.io.util.ColumnIterator$1.getKey(ColumnSortedMap.java=
:276)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.io.util.ColumnIterator$1.getKey(ColumnSortedMap.java=
:263)
> =A0 =A0 =A0 =A0at
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(Unknown Source=
)
> =A0 =A0 =A0 =A0at java.util.concurrent.ConcurrentSkipListMap.<init>(Unkno=
wn Source)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.jav=
a:384)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.jav=
a:332)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnF=
amilySerializer.java:129)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWi=
thColumns(SSTableIdentityIterator.java:137)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIter=
ator.java:139)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.=
java:108)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.=
java:43)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.=
java:73)
> =A0 =A0 =A0 =A0at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera=
tor.java:136)
> =A0 =A0 =A0 =A0at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:=
131)
> =A0 =A0 =A0 =A0at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(Fil=
terIterator.java:183)
> =A0 =A0 =A0 =A0at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIte=
rator.java:94)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.=
java:427)
> =A0 =A0 =A0 =A0at
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:2=
17)
> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerRun(Unknown S=
ource)
> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.run(Unknown Source)
> =A0 =A0 =A0 =A0... 3 more
>
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.=
3065146.n2.nabble.com/Java-bombs-during-compaction-please-help-tp6001773p60=
01773.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at=
 Nabble.com.
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com