cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9686) FSReadError and LEAK DETECTED after upgrading
Date Wed, 08 Jul 2015 08:26:04 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14618207#comment-14618207
] 

Stefania commented on CASSANDRA-9686:
-------------------------------------

This is ready for review.

I added handling of {{CorruptSSTableException}} and {{FSError}} when loading sstables. When
the disk failure policy is {{die}}, the process will exit as expected. For {{CorruptSSTableException}},
the transports will be stopped only with {{stop_paranoid}}, not just {{stop}}, as per documentation
in the yaml file. 

One potential issue is that tables are loaded during the initial health checks, before the
transports are started. So stopping the transports has no effect. Later on they are started
as if nothing happened. So effectively only the {{die}} policy is honored in this scenario.
Not sure what to do about this as it could be legit to want to restart the transports later
on via JMX.

CI results:

http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9686-2.2-testall/lastCompletedBuild/testReport/
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9686-2.2-dtest/lastCompletedBuild/testReport/

One flacky dtest, I've scheduled a new build (#4).

> FSReadError and LEAK DETECTED after upgrading
> ---------------------------------------------
>
>                 Key: CASSANDRA-9686
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9686
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Windows-7-32 bit, 3.2GB RAM, Java 1.7.0_55
>            Reporter: Andreas Schnitzerling
>            Assignee: Stefania
>             Fix For: 2.2.x
>
>         Attachments: cassandra.bat, cassandra.yaml, compactions_in_progress.zip, sstable_activity.zip,
system.log
>
>
> After upgrading one of 15 nodes from 2.1.7 to 2.2.0-rc1 I get FSReadError and LEAK DETECTED
on start. Deleting the listed files, the failure goes away.
> {code:title=system.log}
> ERROR [SSTableBatchOpen:1] 2015-06-29 14:38:34,554 DebuggableThreadPoolExecutor.java:242
- Error in ThreadPoolExecutor
> org.apache.cassandra.io.FSReadError: java.io.IOException: Compressed file with 0 chunks
encountered: java.io.DataInputStream@1c42271
> 	at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:178)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:117)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:178)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:681)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:644)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:443)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:350)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:480)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_55]
> 	at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_55]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_55]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_55]
> 	at java.lang.Thread.run(Unknown Source) [na:1.7.0_55]
> Caused by: java.io.IOException: Compressed file with 0 chunks encountered: java.io.DataInputStream@1c42271
> 	at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:174)
~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> 	... 15 common frames omitted
> ERROR [Reference-Reaper:1] 2015-06-29 14:38:34,734 Ref.java:189 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3e547f) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1926439:D:\Programme\Cassandra\data\data\system\compactions_in_progress\system-compactions_in_progress-ka-6866
was not released before the reference was garbage collected
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message