Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 2862E200C45 for ; Tue, 28 Mar 2017 23:45:47 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 270D9160B89; Tue, 28 Mar 2017 21:45:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6DB0C160B6B for ; Tue, 28 Mar 2017 23:45:46 +0200 (CEST) Received: (qmail 89432 invoked by uid 500); 28 Mar 2017 21:45:45 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 89420 invoked by uid 99); 28 Mar 2017 21:45:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Mar 2017 21:45:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 31F15C0587 for ; Tue, 28 Mar 2017 21:45:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id zZpTeQhJUXXa for ; Tue, 28 Mar 2017 21:45:44 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 9918B5FDB1 for ; Tue, 28 Mar 2017 21:45:43 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 06326E06BA for ; Tue, 28 Mar 2017 21:45:43 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 014C325CF1 for ; Tue, 28 Mar 2017 21:45:42 +0000 (UTC) Date: Tue, 28 Mar 2017 21:45:42 +0000 (UTC) From: "Jeff Jirsa (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-12728) Handling partially written hint files MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 28 Mar 2017 21:45:47 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946034#comment-15946034 ] Jeff Jirsa commented on CASSANDRA-12728: ---------------------------------------- Seems like a bug fix to me, would hope for 3.0 and newer personally. > Handling partially written hint files > ------------------------------------- > > Key: CASSANDRA-12728 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12728 > Project: Cassandra > Issue Type: Bug > Reporter: Sharvanath Pathak > Assignee: Garvit Juniwal > Labels: lhf > Attachments: CASSANDRA-12728.patch > > > {noformat} > ERROR [HintsDispatcher:1] 2016-09-28 17:44:43,397 HintsDispatchExecutor.java:225 - Failed to dispatch hints file d5d7257c-9f81-49b2-8633-6f9bda6e3dea-1474892654160-1.hints: file is corrupted ({}) > org.apache.cassandra.io.FSReadError: java.io.EOFException > at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259) [apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242) [apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220) [apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199) [apache-cassandra-3.0.6.jar:3.0.6] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] > Caused by: java.io.EOFException: null > at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.ChecksummedDataInput.readFully(ChecksummedDataInput.java:126) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsReader$BuffersIterator.readBuffer(HintsReader.java:310) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:301) ~[apache-cassandra-3.0.6.jar:3.0.6] > at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:278) ~[apache-cassandra-3.0.6.jar:3.0.6] > ... 15 common frames omitted > {noformat} > We've found out that the hint file was truncated because there was a hard reboot around the time of last write to the file. I think we basically need to handle partially written hint files. Also, the CRC file does not exist in this case (probably because it crashed while writing the hints file). May be ignoring and cleaning up such partially written hint files can be a way to fix this? -- This message was sent by Atlassian JIRA (v6.3.15#6346)