cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefano Ortolani (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
Date Tue, 24 May 2016 17:06:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298522#comment-15298522
] 

Stefano Ortolani commented on CASSANDRA-11470:
----------------------------------------------

I am not sure it's related, but I noticed another exception while checking my logs. 
Note that, unlike with CASSANDRA-11823, it's not related to creating a new column family.

{code:xml}
WARN  [CompactionExecutor:2006] 2016-05-23 18:23:31,694 BigTableWriter.java:171 - Writing
large partition test_keyspace/test_columnfamily:eda6b9c36f8df6fe596492c3438d7a38e9b109a6 (123663388
bytes)
INFO  [IndexSummaryManager:1] 2016-05-23 18:24:23,731 IndexSummaryRedistribution.java:74 -
Redistributing index summaries
WARN  [CompactionExecutor:2006] 2016-05-23 18:24:56,669 BigTableWriter.java:171 - Writing
large partition test_keyspace/test_columnfamily:05b6b424194dd19ab7cfbcd53c4979768cd859e9 (256286063
bytes)
WARN  [CompactionExecutor:2006] 2016-05-23 18:26:23,575 BigTableWriter.java:171 - Writing
large partition test_keyspace/test_columnfamily:04e9fac15552b9ae77c27a6cb8d0fd11fdcc24d7 (212445557
bytes)
INFO  [CompactionExecutor:2005] 2016-05-23 18:29:26,839 LeveledManifest.java:437 - Adding
high-level (L3) BigTableReader(path='/data/cassandra/data/test_keyspace/test_columnfamily_2-d29dd71045a811e59aff6776bf484396/ma-61041-big-Data.db')
to candidates
WARN  [CompactionExecutor:2006] 2016-05-23 18:30:34,154 BigTableWriter.java:171 - Writing
large partition test_keyspace/test_columnfamily:edbe6f178503be90911dbf29a55b97a4b095a9ec (183852539
bytes)
INFO  [CompactionExecutor:2006] 2016-05-23 18:31:21,080 LeveledManifest.java:437 - Adding
high-level (L3) BigTableReader(path='/data/cassandra/data/test_keyspace/test_columnfamily_2-d29dd71045a811e59aff6776bf484396/ma-61042-big-Data.db')
to candidates
ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-23 18:31:21,207 LogFile.java:173 - Unexpected
files detected for sstable [ma-91034-big], record [REMOVE:[/data/cassandra/data/test_keyspace/test_columnfamily-3996ce80b7ac11e48a9b6776bf484396/ma-91034-big,1463992176000,8][457420186]]:
last update time [00:00:00] should have been [08:29:36]
ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-23 18:31:21,208 ScheduledReporter.java:119
- RuntimeException thrown from GraphiteReporter#report. Exception was suppressed.
java.lang.RuntimeException: Failed to list files in /data/cassandra/data/test_keyspace/test_columnfamily-3996ce80b7ac11e48a9b6776bf484396
	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:57)
~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:691) ~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.db.Directories$SSTableLister.listFiles(Directories.java:662) ~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.db.Directories$TrueFilesSizeVisitor.<init>(Directories.java:981)
~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.db.Directories.getTrueAllocatedSizeIn(Directories.java:893) ~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.db.Directories.trueSnapshotsSize(Directories.java:883) ~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.db.ColumnFamilyStore.trueSnapshotsSize(ColumnFamilyStore.java:2332)
~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:637) ~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:634) ~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.metrics.TableMetrics$33.getValue(TableMetrics.java:692) ~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.metrics.TableMetrics$33.getValue(TableMetrics.java:686) ~[apache-cassandra-3.0.6.jar:3.0.6]
	at com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:281)
~[metrics-graphite-3.1.0.jar:3.1.0]
	at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158) ~[metrics-graphite-3.1.0.jar:3.1.0]
	at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) ~[metrics-core-3.1.0.jar:3.1.0]
	at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) ~[metrics-core-3.1.0.jar:3.1.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_91]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_91]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.cassandra.db.lifecycle.LogTransaction$CorruptTransactionLogException:
Some records failed verification. See earlier in log for details.
	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.readTxnLog(LogAwareFileLister.java:114)
~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:106)
~[apache-cassandra-3.0.6.jar:3.0.6]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_91]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_91]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[na:1.8.0_91]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_91]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_91]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_91]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_91]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_91]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_91]
	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:75)
~[apache-cassandra-3.0.6.jar:3.0.6]
	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
~[apache-cassandra-3.0.6.jar:3.0.6]
	... 22 common frames omitted
INFO  [CompactionExecutor:2005] 2016-05-23 18:32:27,284 LeveledManifest.java:437 - Adding
high-level (L3) BigTableReader(path='/data/cassandra/data/test_keyspace/test_columnfamily_3-37a9cb90b7ac11e48a9b6776bf484396/ma-98189-big-Data.db')
to candidates
INFO  [CompactionExecutor:2005] 2016-05-23 18:34:42,562 LeveledManifest.java:437 - Adding
high-level (L3) BigTableReader(path='/data/cassandra/data/test_keyspace/test_columnfamily_3-37a9cb90b7ac11e48a9b6776bf484396/ma-98210-big-Data.db')
to candidates
{code}


> dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-11470
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Philip Thompson
>            Assignee: Stefania
>              Labels: dtest
>             Fix For: 3.6, 3.0.6
>
>         Attachments: node1.log, node2.log, node2_debug.log, node3.log, node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in the log
of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception encountered
during startup
> java.lang.RuntimeException: Failed to list files in /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
>         at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
~[main/:na]
>         at org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
~[main/:na]
>         at org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
~[main/:na]
>         at org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
~[main/:na]
>         at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
~[main/:na]
>         at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
~[main/:na]
>         at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383) ~[main/:na]
>         at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:320) ~[main/:na]
>         at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130) ~[main/:na]
>         at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107) ~[main/:na]
>         at org.apache.cassandra.cql3.restrictions.StatementRestrictions.<init>(StatementRestrictions.java:139)
~[main/:na]
>         at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
~[main/:na]
>         at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
~[main/:na]
>         at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
~[main/:na]
>         at org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
~[main/:na]
>         at org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
~[main/:na]
>         at org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
~[main/:na]
>         at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
~[main/:na]
>         at org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
~[main/:na]
>         at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
~[main/:na]
>         at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
~[main/:na]
>         at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134) ~[main/:na]
>         at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124) ~[main/:na]
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
[main/:na]
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
[main/:na]
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
[main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
inconsistent disk state for transaction [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log
in /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
>         at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
~[main/:na]
>         at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
~[main/:na]
>         at org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
Source) ~[na:na]
>         at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_45]
>         at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
~[na:1.8.0_45]
>         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
~[na:1.8.0_45]
>         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512) ~[na:1.8.0_45]
>         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
~[na:1.8.0_45]
>         at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
~[na:1.8.0_45]
>         at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
~[na:1.8.0_45]
>         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_45]
>         at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_45]
>         at org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:71)
~[main/:na]
>         at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:49)
~[main/:na]
>         ... 25 common frames omitted
> {code}
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1092/testReport/materialized_views_test/TestMaterializedViews/base_replica_repair_test
> Failed on CassCI build trunk_dtest #1092
> I've attached the logs from the failure in build 1092.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message