cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan Podkowinski (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12762) Cassandra 3.0.9 Fails both compact and repair without even debug logs
Date Tue, 14 Feb 2017 09:00:50 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865395#comment-15865395
] 

Stefan Podkowinski commented on CASSANDRA-12762:
------------------------------------------------

Logging for CorruptBlockException has already been improved in CASSANDRA-12889. You'd have
to upgrade to 3.0.10 for that.

> Cassandra 3.0.9 Fails both compact and repair without even debug logs
> ---------------------------------------------------------------------
>
>                 Key: CASSANDRA-12762
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12762
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Debian Jessie current
>            Reporter: Jason Kania
>            Priority: Critical
>
> After upgrading from 3.0.7 to 3.0.9, the following exception occurs when trying to run
compaction (previous to the upgrade compaction worked fine):
> {code}
> error: (/home/circuitwatch/cassandra/data/circuitwatch/edgeTransitionByCircuitId-f5d33310024b11e5bb310d2316086bf7/mb-12063-big-Data.db):
corruption detected, chunk at 345885546 of length 62024.
> -- StackTrace --
> org.apache.cassandra.io.compress.CorruptBlockException: (/home/circuitwatch/cassandra/data/circuitwatch/edgeTransitionByCircuitId-f5d33310024b11e5bb310d2316086bf7/mb-12063-big-Data.db):
corruption detected, chunk at 345885546 of length 62024.
>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:202)
>         at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
>         at org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
>         at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
>         at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>         at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:404)
>         at org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:406)
>         at org.apache.cassandra.db.rows.BufferCell$Serializer.deserialize(BufferCell.java:302)
>         at org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:476)
>         at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:454)
>         at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:377)
>         at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:87)
>         at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:65)
>         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>         at org.apache.cassandra.io.sstable.SSTableIdentityIterator.doCompute(SSTableIdentityIterator.java:123)
>         at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
>         at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
>         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>         at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
>         at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>         at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
>         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>         at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:509)
>         at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:369)
>         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>         at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
>         at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111)
>         at org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52)
>         at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:149)
>         at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
>         at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
>         at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
>         at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>         at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:606)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> {code}
> Seeing that there was a corruption, I ran nodetool repair, but it also fails in a spectacularly
poor manner giving no indication of the problem:
> {code}
> error: Repair job has failed with the error message: [2016-10-08 01:25:22,431] Some repair
failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message: [2016-10-08
01:25:22,431] Some repair failed
>         at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)
>         at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
> {code}
> The debug logs contain nothing and are just cut at the time of the error. Please update
the errors in the repair case to communicate something meaningful as these sorts of errors
mean the user is stranded.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message