cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Petrov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13229) dtest failure in topology_test.TestTopology.size_estimates_multidc_test
Date Fri, 31 Mar 2017 12:20:42 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950774#comment-15950774
] 

Alex Petrov commented on CASSANDRA-13229:
-----------------------------------------

bq. If there are more disks than vnode ranges, I think we should fall back to splitting vnodes
over the available disks, otherwise we would leave some disks unused

But wouldn't this lead to the problems with disk blacklisting and resurrecting data like Paulo
said in [CASSANDRA-6696]?.. If not, we could go with "try dividing with {{splitOwnedRangesNoPartialRanges}}
first, if some disks are still unutilized, fall back to splitting vnode ranges. 

> dtest failure in topology_test.TestTopology.size_estimates_multidc_test
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-13229
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13229
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Testing
>            Reporter: Sean McCarthy
>            Assignee: Alex Petrov
>              Labels: dtest, test-failure
>             Fix For: 4.0
>
>         Attachments: node1_debug.log, node1_gc.log, node1.log, node2_debug.log, node2_gc.log,
node2.log, node3_debug.log, node3_gc.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/508/testReport/topology_test/TestTopology/size_estimates_multidc_test
> {code}
> Standard Output
> Unexpected error in node1 log, error: 
> ERROR [MemtablePostFlush:1] 2017-02-15 16:07:33,837 CassandraDaemon.java:211 - Exception
in thread Thread[MemtablePostFlush:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> 	at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> 	at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
~[main/:na]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59) ~[main/:na]
> 	at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
~[main/:na]
> 	at org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312) ~[main/:na]
> 	at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304) ~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
~[main/:na]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
[main/:na]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
Source) [main/:na]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Unexpected error in node1 log, error: 
> ERROR [MigrationStage:1] 2017-02-15 16:07:33,853 CassandraDaemon.java:211 - Exception
in thread Thread[MigrationStage:1,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 3, Size: 3
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401) ~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284)
~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown
Source) ~[na:na]
> 	at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
> 	at org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284) ~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265)
~[main/:na]
> 	at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
> 	at org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564) ~[main/:na]
> 	at org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402)
~[main/:na]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
[main/:na]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
Source) [main/:na]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 3, Size: 3
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_45]
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_45]
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[main/:na]
> 	... 16 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> 	at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> 	at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
~[main/:na]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59) ~[main/:na]
> 	at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
~[main/:na]
> 	at org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312) ~[main/:na]
> 	at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304) ~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
~[main/:na]
> 	... 5 common frames omitted
> Unexpected error in node1 log, error: 
> ERROR [main] 2017-02-15 16:07:33,857 CassandraDaemon.java:663 - Exception encountered
during startup
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 3, Size:
3
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401) ~[main/:na]
> 	at org.apache.cassandra.schema.MigrationManager.announce(MigrationManager.java:384)
~[main/:na]
> 	at org.apache.cassandra.schema.MigrationManager.announceNewKeyspace(MigrationManager.java:176)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.maybeAddKeyspace(StorageService.java:1066)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.maybeAddOrUpdateKeyspace(StorageService.java:1091)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1048)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1043)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:966)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:649) ~[main/:na]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581) ~[main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:364) [main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557) [main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:646) [main/:na]
> Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_45]
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_45]
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[main/:na]
> 	... 12 common frames omitted
> Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 3, Size: 3
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401) ~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284)
~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown
Source) ~[na:na]
> 	at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
> 	at org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284) ~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265)
~[main/:na]
> 	at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
> 	at org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564) ~[main/:na]
> 	at org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402)
~[main/:na]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_45]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
~[main/:na]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
Source) ~[na:na]
> 	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> Caused by: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 3, Size: 3
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_45]
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_45]
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[main/:na]
> 	... 16 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> 	at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> 	at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
~[main/:na]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59) ~[main/:na]
> 	at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
~[main/:na]
> 	at org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312) ~[main/:na]
> 	at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304) ~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
~[main/:na]
> 	... 5 common frames omitted
> Unexpected error in node1 log, error: 
> ERROR [StorageServiceShutdownHook] 2017-02-15 16:07:35,972 AbstractCommitLogSegmentManager.java:311
- Failed to force-recycle all segments; at least one segment is still in use with dirty CFs.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message