cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13229) dtest failure in topology_test.TestTopology.size_estimates_multidc_test
Date Fri, 31 Mar 2017 15:53:41 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951175#comment-15951175
] 

Paulo Motta commented on CASSANDRA-13229:
-----------------------------------------

Thanks for the feedback Marcus!

bq. But wouldn't this lead to the problems with disk blacklisting and resurrecting data like
Paulo said in CASSANDRA-6696?

There's no actual problem from what Marcus said, we just wanted to keep vnodes tied to a single
disk to be able to take vnodes offline if a specific disk fails but this hasn't been implemented
yet. Furthermore CASSANDRA-10540 will probably benefit if vnodes are kept in a single disk.

With this said, we should probably keep your approach of falling back to split evenly if not
all disks are used and also raise the minimum number of tokens to split by vnode from 2 to
8 to ensure a better balance (Marcus suggested 16 but 8 should be good already since there
are 24 local ranges with RF=3 and CASSANDRA-7032 should also improve load balancing). We can
probably reconsider this if we ever implement the optimization of blacklisting specific vnodes
when disks fail.

In your initial approach, I'd just move the fallback out of {{splitOwnedRanges}} because it's
a bit wrong to pass {{dontSplitRanges=true}} and have your ranges splitted. :-)

> dtest failure in topology_test.TestTopology.size_estimates_multidc_test
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-13229
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13229
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Testing
>            Reporter: Sean McCarthy
>            Assignee: Alex Petrov
>              Labels: dtest, test-failure
>             Fix For: 4.0
>
>         Attachments: node1_debug.log, node1_gc.log, node1.log, node2_debug.log, node2_gc.log,
node2.log, node3_debug.log, node3_gc.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/508/testReport/topology_test/TestTopology/size_estimates_multidc_test
> {code}
> Standard Output
> Unexpected error in node1 log, error: 
> ERROR [MemtablePostFlush:1] 2017-02-15 16:07:33,837 CassandraDaemon.java:211 - Exception
in thread Thread[MemtablePostFlush:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> 	at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> 	at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
~[main/:na]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59) ~[main/:na]
> 	at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
~[main/:na]
> 	at org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312) ~[main/:na]
> 	at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304) ~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
~[main/:na]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
[main/:na]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
Source) [main/:na]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Unexpected error in node1 log, error: 
> ERROR [MigrationStage:1] 2017-02-15 16:07:33,853 CassandraDaemon.java:211 - Exception
in thread Thread[MigrationStage:1,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 3, Size: 3
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401) ~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284)
~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown
Source) ~[na:na]
> 	at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
> 	at org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284) ~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265)
~[main/:na]
> 	at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
> 	at org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564) ~[main/:na]
> 	at org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402)
~[main/:na]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
[main/:na]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
Source) [main/:na]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 3, Size: 3
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_45]
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_45]
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[main/:na]
> 	... 16 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> 	at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> 	at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
~[main/:na]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59) ~[main/:na]
> 	at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
~[main/:na]
> 	at org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312) ~[main/:na]
> 	at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304) ~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
~[main/:na]
> 	... 5 common frames omitted
> Unexpected error in node1 log, error: 
> ERROR [main] 2017-02-15 16:07:33,857 CassandraDaemon.java:663 - Exception encountered
during startup
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 3, Size:
3
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401) ~[main/:na]
> 	at org.apache.cassandra.schema.MigrationManager.announce(MigrationManager.java:384)
~[main/:na]
> 	at org.apache.cassandra.schema.MigrationManager.announceNewKeyspace(MigrationManager.java:176)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.maybeAddKeyspace(StorageService.java:1066)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.maybeAddOrUpdateKeyspace(StorageService.java:1091)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1048)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1043)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:966)
~[main/:na]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:649) ~[main/:na]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581) ~[main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:364) [main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557) [main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:646) [main/:na]
> Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_45]
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_45]
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[main/:na]
> 	... 12 common frames omitted
> Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 3, Size: 3
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401) ~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284)
~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown
Source) ~[na:na]
> 	at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
> 	at org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284) ~[main/:na]
> 	at org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265)
~[main/:na]
> 	at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
> 	at org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564) ~[main/:na]
> 	at org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402)
~[main/:na]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_45]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
~[main/:na]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
Source) ~[na:na]
> 	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> Caused by: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 3, Size: 3
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_45]
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_45]
> 	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[main/:na]
> 	... 16 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> 	at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> 	at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
~[main/:na]
> 	at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59) ~[main/:na]
> 	at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
~[main/:na]
> 	at org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312) ~[main/:na]
> 	at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304) ~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
~[main/:na]
> 	at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
~[main/:na]
> 	... 5 common frames omitted
> Unexpected error in node1 log, error: 
> ERROR [StorageServiceShutdownHook] 2017-02-15 16:07:35,972 AbstractCommitLogSegmentManager.java:311
- Failed to force-recycle all segments; at least one segment is still in use with dirty CFs.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message