cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksandr Ivanov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12728) Handling partially written hint files
Date Tue, 28 Nov 2017 19:23:01 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269314#comment-16269314
] 

Aleksandr Ivanov commented on CASSANDRA-12728:
----------------------------------------------

Seems it is not fully fixed. Met it in v3.0.14

INFO  [main] 2017-11-28 19:10:13,554 StorageService.java:575 - Cassandra version: 3.0.14
INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:576 - Thrift API version: 20.1.0
INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:577 - CQL supported versions: 3.4.0
(default: 3.4.0)
ERROR [main] 2017-11-28 19:10:13,592 CassandraDaemon.java:710 - Exception encountered during
startup
org.apache.cassandra.io.FSReadError: java.io.EOFException
        at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:142)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_141]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_141]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_141]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
~[na:1.8.0_141]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_141]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_141]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_141]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_141]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_141]
        at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.hints.HintsService.<init>(HintsService.java:88) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.hints.HintsService.<clinit>(HintsService.java:63) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.StorageProxy.<clinit>(StorageProxy.java:121)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at java.lang.Class.forName0(Native Method) ~[na:1.8.0_141]
        at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_141]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:570)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346) [apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) [apache-cassandra-3.0.14.jar:3.0.14]
Caused by: java.io.EOFException: null
        at java.io.RandomAccessFile.readInt(RandomAccessFile.java:803) ~[na:1.8.0_141]
        at org.apache.cassandra.hints.HintsDescriptor.deserialize(HintsDescriptor.java:237)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:138)
~[apache-cassandra-3.0.14.jar:3.0.14]
        ... 20 common frames omitted


After several 0 size hints files deletion Cassandra started successfully. 

> Handling partially written hint files
> -------------------------------------
>
>                 Key: CASSANDRA-12728
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12728
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Sharvanath Pathak
>            Assignee: Garvit Juniwal
>              Labels: lhf
>             Fix For: 3.0.14, 3.11.0, 4.0
>
>         Attachments: CASSANDRA-12728.patch
>
>
> {noformat}
> ERROR [HintsDispatcher:1] 2016-09-28 17:44:43,397 HintsDispatchExecutor.java:225 - Failed
to dispatch hints file d5d7257c-9f81-49b2-8633-6f9bda6e3dea-1474892654160-1.hints: file is
corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
>         at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
[apache-cassandra-3.0.6.jar:3.0.6]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_77]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_77]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> Caused by: java.io.EOFException: null
>         at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.ChecksummedDataInput.readFully(ChecksummedDataInput.java:126)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) ~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsReader$BuffersIterator.readBuffer(HintsReader.java:310)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:301)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:278)
~[apache-cassandra-3.0.6.jar:3.0.6]
>         ... 15 common frames omitted
> {noformat}
> We've found out that the hint file was truncated because there was a hard reboot around
the time of last write to the file. I think we basically need to handle partially written
hint files. Also, the CRC file does not exist in this case (probably because it crashed while
writing the hints file). May be ignoring and cleaning up such partially written hint files
can be a way to fix this?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message