Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (nike.apache.org: 216.139.236.26 is neither permitted
 nor denied by domain of potekhin@bnl.gov)
Date: Mon, 7 Feb 2011 13:02:41 -0800 (PST)
From: buddhasystem <potekhin@bnl.gov>
To: cassandra-user@incubator.apache.org
Message-ID: <1297112561016-6001773.post@n2.nabble.com>
Subject: Java bombs during compaction, please help
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit


Hello,
one node in my 3-machine cluster cannot perform compaction. I tried multiple
times, it ran out of heap space once and I increased it. Now I'm getting the
dump below (after it does run for a few minutes). I hope somebody can shed a
little light on what' going on, because I'm at a loss and this is a real
show stopper.


[me@mymachine]~/cassandra-test% Error occured while compacting keyspace
Tracer
java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
        at java.util.concurrent.FutureTask.get(Unknown Source)
        at
org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
        at
org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1766)
        at
org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1236)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown
Source)
        at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown
Source)
        at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown
Source)
        at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown
Source)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown
Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
        at sun.rmi.transport.Transport$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown
Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown
Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
        at
org.apache.cassandra.io.util.ColumnIterator$1.getKey(ColumnSortedMap.java:276)
        at
org.apache.cassandra.io.util.ColumnIterator$1.getKey(ColumnSortedMap.java:263)
        at
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(Unknown Source)
        at java.util.concurrent.ConcurrentSkipListMap.<init>(Unknown Source)
        at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:384)
        at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:332)
        at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:137)
        at
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
        at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
        at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
        at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
        at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
        at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
        at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
        at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
        at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        ... 3 more


-- 
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Java-bombs-during-compaction-please-help-tp6001773p6001773.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.