Return-Path: Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: (qmail 16916 invoked from network); 10 Mar 2011 15:56:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 10 Mar 2011 15:56:28 -0000 Received: (qmail 85731 invoked by uid 500); 10 Mar 2011 15:56:27 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 85708 invoked by uid 500); 10 Mar 2011 15:56:27 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 85677 invoked by uid 99); 10 Mar 2011 15:56:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 15:56:27 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 15:56:22 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id 6BD013A220E for ; Thu, 10 Mar 2011 15:56:00 +0000 (UTC) Date: Thu, 10 Mar 2011 15:56:00 +0000 (UTC) From: "Mike Nadeau (JIRA)" To: commits@cassandra.apache.org Message-ID: <18635308.10905.1299772560437.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <2017442075.10808.1299769920006.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005119#comment-13005119 ] Mike Nadeau commented on CASSANDRA-2307: ---------------------------------------- No luck :( I'm still getting this error during the compaction: Error occured during compaction java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:209) at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1720) at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1263) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source) at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) at sun.rmi.transport.Transport$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Unknown Source) at java.nio.HeapByteBuffer.getInt(Unknown Source) at org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57) at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:852) at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:839) at org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:830) at org.apache.cassandra.io.PrecompactedRow.(PrecompactedRow.java:94) at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139) at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108) at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449) at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:240) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) ... 3 more I'll be happy to provide SSTables, I just don't know which files would be helpful. > IndexOutOfBoundsException during compaction > ------------------------------------------- > > Key: CASSANDRA-2307 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2307 > Project: Cassandra > Issue Type: Bug > Affects Versions: 0.7.2, 0.7.3 > Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned to Cassandra, JRE 1.6.0_24 > Reporter: Mike Nadeau > > We're getting an IndexOutOfBounds exception when compacting. > Here's the detailed error we get on-screen when running "nodetool -h 10.3.133.10 compact": > Error occured while compacting keyspace test > java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException > at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) > at java.util.concurrent.FutureTask.get(Unknown Source) > at org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186) > at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678) > at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) > at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) > at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source) > at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source) > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source) > at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source) > at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) > at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source) > at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source) > at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source) > at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source) > at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) > at sun.rmi.transport.Transport$1.run(Unknown Source) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Unknown Source) > at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.IndexOutOfBoundsException > at java.nio.Buffer.checkIndex(Unknown Source) > at java.nio.HeapByteBuffer.getInt(Unknown Source) > at org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57) > at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822) > at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809) > at org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800) > at org.apache.cassandra.io.PrecompactedRow.(PrecompactedRow.java:94) > at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139) > at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108) > at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43) > at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73) > at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) > at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) > at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) > at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) > at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427) > at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217) > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > ... 3 more > And here's the error I'm getting in my log file: > ERROR [CompactionExecutor:1] 2011-03-09 19:16:52,299 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] > java.lang.IndexOutOfBoundsException > at java.nio.Buffer.checkIndex(Unknown Source) > at java.nio.HeapByteBuffer.getInt(Unknown Source) > at org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57) > at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822) > at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809) > at org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800) > at org.apache.cassandra.io.PrecompactedRow.(PrecompactedRow.java:94) > at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139) > at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108) > at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43) > at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73) > at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) > at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) > at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) > at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) > at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427) > at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217) > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > We run Cassandra 0.7.3, but we had the issue with Cassandra 0.7.2 as well. We have 8 machines in the cluster, the error happens only on one machine, at least for now. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira