cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruggero Marchei (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10375) nodetool repair fails
Date Sat, 19 Sep 2015 12:33:04 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14877084#comment-14877084
] 

Ruggero Marchei commented on CASSANDRA-10375:
---------------------------------------------

I think I hit [#10057|CASSANDRA-10057]. I'm closing this one as duplicated

> nodetool repair fails
> ---------------------
>
>                 Key: CASSANDRA-10375
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10375
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core, Tools
>         Environment: multi-dc Cassandra 2.2.1 cluster with 6 nodes, 3 per DC
>            Reporter: Ruggero Marchei
>             Fix For: 2.2.2
>
>
> When I'm running a *nodetool repair* it gets stalled after few seconds:
> {code}[2015-09-19 11:12:13,807] Repair session 479ca1c0-5ebf-11e5-9619-3f4813058061 for
range (40511972970986385,59154612555757611] failed with error [repair #479ca1c0-5ebf-11e5-9619-3f4813058061
on static_assets/assets, (40511972970986385,59154612555757611]] Validation failed in /10.8.34.113
(progress: 0%)
> [2015-09-19 11:12:13,812] Repair session 479cc8d1-5ebf-11e5-9619-3f4813058061 for range
(6553929828848556033,6576029219234973671] failed with error [repair #479cc8d1-5ebf-11e5-9619-3f4813058061
on static_assets/assets, (6553929828848556033,6576029219234973671]] Validation failed in /10.8.34.113
(progress: 0%)
> {code}
> At the same time I have this exception on another node (I'm not running multiple repairs):
> {code}
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,825 CompactionManager.java:1070 - Cannot
start multiple repair sessions over the same sstables
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,826 Validator.java:246 - Failed creating
a merkle tree for [repair #479c2c90-5ebf-11e5-9619-3f4813058061 on static_assets/assets, (-2926621365236563900,-2916361392298929067]],
/10.8.34.113 (see log for details)
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,826 CassandraDaemon.java:183 - Exception
in thread Thread[ValidationExecutor:66,1,main]
> java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables
>         at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,900 CompactionManager.java:1070 - Cannot
start multiple repair sessions over the same sstables
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,900 Validator.java:246 - Failed creating
a merkle tree for [repair #479c53a1-5ebf-11e5-9619-3f4813058061 on static_assets/assets, (8236929501578674892,8238760988019827700]],
/10.8.34.113 (see log for details)
> ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,900 CompactionManager.java:1070 - Cannot
start multiple repair sessions over the same sstables
> ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,900 CompactionManager.java:1070 - Cannot
start multiple repair sessions over the same sstables
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,901 CassandraDaemon.java:183 - Exception
in thread Thread[ValidationExecutor:66,1,main]
> java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables
>         at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,901 Validator.java:246 - Failed creating
a merkle tree for [repair #479cc8d1-5ebf-11e5-9619-3f4813058061 on static_assets/assets, (6553929828848556033,6576029219234973671]],
/10.8.34.113 (see log for details)
> ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,901 Validator.java:246 - Failed creating
a merkle tree for [repair #479ca1c0-5ebf-11e5-9619-3f4813058061 on static_assets/assets, (40511972970986385,59154612555757611]],
/10.8.34.113 (see log for details)
> ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,901 CassandraDaemon.java:183 - Exception
in thread Thread[ValidationExecutor:68,1,main]
> java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables
>         at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,901 CassandraDaemon.java:183 - Exception
in thread Thread[ValidationExecutor:67,1,main]
> java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables
>         at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
~[apache-cassandra-2.2.1.jar:2.2.1]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> {code}
> And on the remaining nodes I have something similar to this:
> {code}
> ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@320d7d92) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@780334113:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-2447-big
was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@972fce9) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@389940435:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-6-big
was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@5a2a0617) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@156533865:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-4-big
was not released before the reference was garbage collected
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message