cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Igor Zubchenok (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-14224) Failed to create merkel tree
Date Thu, 08 Feb 2018 22:52:00 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Igor Zubchenok updated CASSANDRA-14224:
---------------------------------------
    Description: 
While repairing a node with nodetool -full -pr -dcpar, another node throws an AssertionError
and failed to create merkel tree.

There is no information what is wrong and how to fix in logs.

 
{code:java}
DEBUG [ValidationExecutor:25] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big
(1.876KiB)
DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big
(1.876KiB)
DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,586 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-36-big
(0.688KiB)
ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,586 Validator.java:268 - Failed creating
a merkle tree for [repair #69803961-0d17-11e8-950c-37c5a1498597 on prod/invoices, [(-4165282778092010776,-4162024061639141985]]],
/88.99.35.178 (see log for details)
ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,587 CassandraDaemon.java:228 - Exception
in thread Thread[ValidationExecutor:25,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.get(SSTableReader.java:2313)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.setup(SSTableReader.java:2138)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.SSTableReader.setup(SSTableReader.java:2081) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:519) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ColumnFamilyStore.getSnapshotSSTableReader(ColumnFamilyStore.java:1901)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1335)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:86)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:942)
~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,588 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-33-big
(9.891MiB)
ERROR [NonPeriodicTasks:1] 2018-02-08 21:31:42,589 CassandraDaemon.java:228 - Exception in
thread Thread[NonPeriodicTasks:1,5,main]
java.lang.NullPointerException: null
at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:2192)
~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
~[na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]{code}

  was:
While repairing a node with nodetool -full -pr -dcpar, another node throws an assertion exception
and failed to create merkel tree:

 
{code:java}
DEBUG [ValidationExecutor:25] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big
(1.876KiB)
DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big
(1.876KiB)
DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,586 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-36-big
(0.688KiB)
ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,586 Validator.java:268 - Failed creating
a merkle tree for [repair #69803961-0d17-11e8-950c-37c5a1498597 on prod/invoices, [(-4165282778092010776,-4162024061639141985]]],
/88.99.35.178 (see log for details)
ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,587 CassandraDaemon.java:228 - Exception
in thread Thread[ValidationExecutor:25,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.get(SSTableReader.java:2313)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.setup(SSTableReader.java:2138)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.SSTableReader.setup(SSTableReader.java:2081) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:519) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ColumnFamilyStore.getSnapshotSSTableReader(ColumnFamilyStore.java:1901)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1335)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:86)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:942)
~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,588 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-33-big
(9.891MiB)
ERROR [NonPeriodicTasks:1] 2018-02-08 21:31:42,589 CassandraDaemon.java:228 - Exception in
thread Thread[NonPeriodicTasks:1,5,main]
java.lang.NullPointerException: null
at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:2192)
~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
~[na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]{code}


> Failed to create merkel tree
> ----------------------------
>
>                 Key: CASSANDRA-14224
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14224
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Repair
>            Reporter: Igor Zubchenok
>            Priority: Major
>
> While repairing a node with nodetool -full -pr -dcpar, another node throws an AssertionError
and failed to create merkel tree.
> There is no information what is wrong and how to fix in logs.
>  
> {code:java}
> DEBUG [ValidationExecutor:25] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening
/var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big
(1.876KiB)
> DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening
/var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big
(1.876KiB)
> DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,586 SSTableReader.java:503 - Opening
/var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-36-big
(0.688KiB)
> ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,586 Validator.java:268 - Failed creating
a merkle tree for [repair #69803961-0d17-11e8-950c-37c5a1498597 on prod/invoices, [(-4165282778092010776,-4162024061639141985]]],
/88.99.35.178 (see log for details)
> ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,587 CassandraDaemon.java:228 - Exception
in thread Thread[ValidationExecutor:25,1,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.get(SSTableReader.java:2313)
~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.setup(SSTableReader.java:2138)
~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.io.sstable.format.SSTableReader.setup(SSTableReader.java:2081)
~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:519)
~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.ColumnFamilyStore.getSnapshotSSTableReader(ColumnFamilyStore.java:1901)
~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1335)
~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:86)
~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:942)
~[apache-cassandra-3.11.1.jar:3.11.1]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.1.jar:3.11.1]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
> DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,588 SSTableReader.java:503 - Opening
/var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-33-big
(9.891MiB)
> ERROR [NonPeriodicTasks:1] 2018-02-08 21:31:42,589 CassandraDaemon.java:228 - Exception
in thread Thread[NonPeriodicTasks:1,5,main]
> java.lang.NullPointerException: null
> at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:2192)
~[apache-cassandra-3.11.1.jar:3.11.1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_151]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
~[na:1.8.0_151]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
~[na:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.1.jar:3.11.1]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message