cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11765) dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_AllVersions_Skips_3_0_x_EndsAt_Trunk_HEAD.rolling_upgrade_test
Date Thu, 02 Jun 2016 14:02:59 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312352#comment-15312352
] 

Stefania commented on CASSANDRA-11765:
--------------------------------------

Thank you for the review.

bq. We could probably just run the tasks one after the other instead?

Done. Previously they were running in the compaction executor and it would have been a problem
with a single threaded executor.

bq. And a code comment - in CompactionStrategyManager;

Comment added and typo fixed, thanks for spotting this.

bq. The catch (InterruptedException | ExecutionException e) { JVMStabilityInspector.inspectThrowable(e);
... doesn't work as expected - you'd need to pass ((ExecutionException)e).getCause() to JVMStabilityInspector.
However - I'd prefer to add support for ExecutionException to JVMStabilityInspector since
we have similar usages elsewhere.

It's no longer relevant but, why would it have been a problem given that JVMStabilityInspector.inspect()
recursively calls itself if getClause() is not null?

I've rebased on 3.7 and cherry-picked to trunk:

||3.7||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/11765-3.7]|[patch|https://github.com/stef1927/cassandra/commits/11765]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11765-3.7-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11765-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11765-3.7-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11765-dtest/]|

The tests are currently running on 3.7, if they are OK I will launch them on trunk as well.


The new dtest is [here|https://github.com/stef1927/cassandra-dtest/commits/11765] and it is
included in the CI jobs currently running.


> dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_AllVersions_Skips_3_0_x_EndsAt_Trunk_HEAD.rolling_upgrade_test
> ------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-11765
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11765
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Philip Thompson
>            Assignee: Stefania
>              Labels: dtest
>         Attachments: node1.log, node1_debug.log, node2.log, node2_debug.log, node3.log,
node3_debug.log
>
>
> Example failure:
> http://cassci.datastax.com/view/Parameterized/job/upgrade_tests-all-custom_branch_runs/12/testReport/upgrade_tests.upgrade_through_versions_test/ProtoV3Upgrade_AllVersions_Skips_3_0_x_EndsAt_Trunk_HEAD/rolling_upgrade_test_2/
> The test is seeing a corrupt hint sstable after upgrade from 2.2.5 to 3.6. Relevant stack
trace is
> {code}
> ERROR [main] 2016-05-11 16:22:25,180 CassandraDaemon.java:727 - Exception encountered
during startup
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.io.sstable.CorruptSSTableException:
Corrupted: /mnt/tmp/dtest-X7IReF/test/node1/data2/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-3-big-Data.db
> 	at org.apache.cassandra.hints.LegacyHintsMigrator.forceCompaction(LegacyHintsMigrator.java:119)
~[main/:na]
> 	at org.apache.cassandra.hints.LegacyHintsMigrator.compactLegacyHints(LegacyHintsMigrator.java:108)
~[main/:na]
> 	at org.apache.cassandra.hints.LegacyHintsMigrator.migrate(LegacyHintsMigrator.java:92)
~[main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:321) [main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:581) [main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:710) [main/:na]
> Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.io.sstable.CorruptSSTableException:
Corrupted: /mnt/tmp/dtest-X7IReF/test/node1/data2/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-3-big-Data.db
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_51]
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_51]
> 	at org.apache.cassandra.hints.LegacyHintsMigrator.forceCompaction(LegacyHintsMigrator.java:115)
~[main/:na]
> 	... 5 common frames omitted
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /mnt/tmp/dtest-X7IReF/test/node1/data2/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-3-big-Data.db
> 	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:351)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:265)
~[main/:na]
> 	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[main/:na]
> 	at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:245)
~[main/:na]
> 	at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:463)
~[main/:na]
> 	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[main/:na]
> 	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
~[main/:na]
> 	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
~[main/:na]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:805)
~[main/:na]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_51]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_51]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_51]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_51]
> 	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> Caused by: java.io.EOFException: null
> 	at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
~[main/:na]
> 	at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
~[main/:na]
> 	at org.apache.cassandra.io.util.TrackedDataInputPlus.readFully(TrackedDataInputPlus.java:93)
~[main/:na]
> 	at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) ~[main/:na]
> 	at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:375)
~[main/:na]
> 	at org.apache.cassandra.db.Serializers$1.deserialize(Serializers.java:109) ~[main/:na]
> 	at org.apache.cassandra.db.Serializers$1.deserialize(Serializers.java:89) ~[main/:na]
> 	at org.apache.cassandra.io.sstable.IndexInfo$Serializer.deserialize(IndexInfo.java:135)
~[main/:na]
> 	at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.<init>(RowIndexEntry.java:651)
~[main/:na]
> 	at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.<init>(RowIndexEntry.java:577)
~[main/:na]
> 	at org.apache.cassandra.db.RowIndexEntry$LegacyShallowIndexedEntry.deserialize(RowIndexEntry.java:508)
~[main/:na]
> 	at org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:321)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:310)
~[main/:na]
> 	... 19 common frames omitted
> ERROR [CompactionExecutor:2] 2016-05-11 16:22:25,183 CassandraDaemon.java:213 - Exception
in thread Thread[CompactionExecutor:2,1,main]
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /mnt/tmp/dtest-X7IReF/test/node1/data2/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-3-big-Data.db
> 	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:351)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:265)
~[main/:na]
> 	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[main/:na]
> 	at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:245)
~[main/:na]
> 	at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:463)
~[main/:na]
> 	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[main/:na]
> 	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
~[main/:na]
> 	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
~[main/:na]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:805)
~[main/:na]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_51]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_51]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_51]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> Caused by: java.io.EOFException: null
> 	at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
~[main/:na]
> 	at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
~[main/:na]
> 	at org.apache.cassandra.io.util.TrackedDataInputPlus.readFully(TrackedDataInputPlus.java:93)
~[main/:na]
> 	at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) ~[main/:na]
> 	at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:375)
~[main/:na]
> 	at org.apache.cassandra.db.Serializers$1.deserialize(Serializers.java:109) ~[main/:na]
> 	at org.apache.cassandra.db.Serializers$1.deserialize(Serializers.java:89) ~[main/:na]
> 	at org.apache.cassandra.io.sstable.IndexInfo$Serializer.deserialize(IndexInfo.java:135)
~[main/:na]
> 	at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.<init>(RowIndexEntry.java:651)
~[main/:na]
> 	at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.<init>(RowIndexEntry.java:577)
~[main/:na]
> 	at org.apache.cassandra.db.RowIndexEntry$LegacyShallowIndexedEntry.deserialize(RowIndexEntry.java:508)
~[main/:na]
> 	at org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:321)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:310)
~[main/:na]
> 	... 19 common frames omitted
> {code}
> Logs are attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message