Return-Path: Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: (qmail 40087 invoked from network); 11 Mar 2011 22:04:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 11 Mar 2011 22:04:22 -0000 Received: (qmail 81619 invoked by uid 500); 11 Mar 2011 22:04:22 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 81591 invoked by uid 500); 11 Mar 2011 22:04:22 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 81552 invoked by uid 99); 11 Mar 2011 22:04:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 22:04:22 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 22:04:20 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id A65783A4C69 for ; Fri, 11 Mar 2011 22:03:59 +0000 (UTC) Date: Fri, 11 Mar 2011 22:03:59 +0000 (UTC) From: "Matthew F. Dennis (JIRA)" To: commits@cassandra.apache.org Message-ID: <1820865328.14579.1299881039677.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] Created: (CASSANDRA-2315) mmap segment underflow when building secondary index after joining ring MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 mmap segment underflow when building secondary index after joining ring ----------------------------------------------------------------------- Key: CASSANDRA-2315 URL: https://issues.apache.org/jira/browse/CASSANDRA-2315 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.3 Reporter: Matthew F. Dennis there was a 7 node ring (N0-N6) that was upgraded to 0.7.3 all the data was scrubbed 7 additional nodes were added (N7-N13) with no problems some compactions ran, some things were restarted (but there was no read/write load) after a while 7 more nodes (N14-N20) were added all joined successfully except one that node was pulling data from one of the nodes in the N7-N13 set (i.e. a node that got all it's data from the original nodes) the EC2 instance that was running was completely destroyed, a new instance created and the bootstrap attempted again but still failed {code} INFO [Thread-65] 2011-03-10 17:16:59,448 ColumnFamilyStore.java (line 377) Index build of service_id, complete INFO [Thread-65] 2011-03-10 17:16:59,449 ColumnFamilyStore.java (line 359) Submitting index build of service_id, for data in SSTableReader(path='/mnt/cassandra/data/BackupifyMetadata/GoogleDocsMetadata-f-1-Data.db') INFO [Thread-75] 2011-03-10 17:17:45,757 ColumnFamilyStore.java (line 377) Index build of service_id, complete INFO [Thread-75] 2011-03-10 17:17:45,767 ColumnFamilyStore.java (line 359) Submitting index build of service_id, for data in SSTableReader(path='/mnt/cassandra/data/BackupifyMetadata/GoogleDocsMetadata-f-2-Data.db'), SSTableReader(path='/mnt/cassandra/data/BackupifyMetadata/GoogleDocsMetadata-f-3-Data.db'), SSTableReader(path='/mnt/cassandra/data/BackupifyMetadata/GoogleDocsMetadata-f-4-Data.db'), SSTableReader(path='/mnt/cassandra/data/BackupifyMetadata/GoogleDocsMetadata-f-5-Data.db') INFO [ScheduledTasks:1] 2011-03-10 17:17:46,550 ColumnFamilyStore.java (line 1020) Enqueuing flush of Memtable-GoogleDocsMetadata.736572766963655f6964@1078361799(353352 bytes, 4539 operations) INFO [FlushWriter:1] 2011-03-10 17:17:46,552 Memtable.java (line 157) Writing Memtable-GoogleDocsMetadata.736572766963655f6964@1078361799(353352 bytes, 4539 operations) INFO [FlushWriter:1] 2011-03-10 17:17:46,736 Memtable.java (line 164) Completed flushing /mnt/cassandra/data/BackupifyMetadata/GoogleDocsMetadata.736572766963655f6964-f-1-Data.db (483139 bytes) INFO [Thread-65] 2011-03-10 17:18:04,278 ColumnFamilyStore.java (line 377) Index build of service_id, complete INFO [Thread-65] 2011-03-10 17:18:04,279 ColumnFamilyStore.java (line 359) Submitting index build of service_id, for data in SSTableReader(path='/mnt/cassandra/data/BackupifyMetadata/BloggerMetadata-f-1-Data.db') ERROR [Thread-75] 2011-03-10 17:18:21,781 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[Thread-75,5,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: mmap segment underflow; remaining is 43239499 but 1768777055 requested at org.apache.cassandra.db.ColumnFamilyStore.buildSecondaryIndexes(ColumnFamilyStore.java:375) at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:159) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: mmap segment underflow; remaining is 43239499 but 1768777055 requested at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.ColumnFamilyStore.buildSecondaryIndexes(ColumnFamilyStore.java:365) ... 3 more Caused by: java.lang.AssertionError: mmap segment underflow; remaining is 43239499 but 1768777055 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35) at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79) at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108) at org.apache.commons.collections.iterators.CollatingIterator.anyHasNext(CollatingIterator.java:364) at org.apache.commons.collections.iterators.CollatingIterator.hasNext(CollatingIterator.java:217) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:55) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:118) at org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:142) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1340) at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1158) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1224) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1145) at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:459) at org.apache.cassandra.db.Table.access$200(Table.java:56) at org.apache.cassandra.db.Table$IndexBuilder.build(Table.java:573) at org.apache.cassandra.db.CompactionManager$8.run(CompactionManager.java:892) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [CompactionExecutor:1] 2011-03-10 17:18:21,785 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError: mmap segment underflow; remaining is 43239499 but 1768777055 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35) at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79) at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108) at org.apache.commons.collections.iterators.CollatingIterator.anyHasNext(CollatingIterator.java:364) at org.apache.commons.collections.iterators.CollatingIterator.hasNext(CollatingIterator.java:217) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:55) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:118) at org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:142) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1340) at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1158) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1224) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1145) at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:459) at org.apache.cassandra.db.Table.access$200(Table.java:56) at org.apache.cassandra.db.Table$IndexBuilder.build(Table.java:573) at org.apache.cassandra.db.CompactionManager$8.run(CompactionManager.java:892) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira