Return-Path: <notifications-return-37958-archive-asf-public=cust-asf.ponee.io@accumulo.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id AE1D3200BF5
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Fri, 23 Dec 2016 21:42:00 +0100 (CET)
Received: by cust-asf.ponee.io (Postfix)
	id ACE1D160B0C; Fri, 23 Dec 2016 20:42:00 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 006D4160B1F
	for <archive-asf-public@cust-asf.ponee.io>; Fri, 23 Dec 2016 21:41:59 +0100 (CET)
Received: (qmail 31673 invoked by uid 500); 23 Dec 2016 20:41:58 -0000
Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:notifications-help@accumulo.apache.org>
List-Unsubscribe: <mailto:notifications-unsubscribe@accumulo.apache.org>
List-Post: <mailto:notifications@accumulo.apache.org>
List-Id: <notifications.accumulo.apache.org>
Reply-To: jira@apache.org
Delivered-To: mailing list notifications@accumulo.apache.org
Received: (qmail 31239 invoked by uid 99); 23 Dec 2016 20:41:58 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Dec 2016 20:41:58 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id 707A12C2A66
	for <notifications@accumulo.apache.org>; Fri, 23 Dec 2016 20:41:58 +0000 (UTC)
Date: Fri, 23 Dec 2016 20:41:58 +0000 (UTC)
From: "Landon Davies (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12820930.1429039040000.591250.1482525718457@Atlassian.JIRA>
In-Reply-To: <JIRA.12820930.1429039040000@Atlassian.JIRA>
References: <JIRA.12820930.1429039040000@Atlassian.JIRA> <JIRA.12820930.1429039040858@arcas>
Subject: [jira] [Commented] (ACCUMULO-3727) FileNotFoundException on
 failed/data during recovery
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
archived-at: Fri, 23 Dec 2016 20:42:00 -0000


    [ https://issues.apache.org/jira/browse/ACCUMULO-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15773653#comment-15773653 ] 

Landon Davies commented on ACCUMULO-3727:
-----------------------------------------

I encountered this same problem on an Accumulo 1.6.0 instance. It does indeed look like a failed recovery, then reassignment scenario

Resolved it manually by removing the file hdfs:/accumulo/recovery/<ID>/failed


> FileNotFoundException on failed/data during recovery
> ----------------------------------------------------
>
>                 Key: ACCUMULO-3727
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3727
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.5.2
>            Reporter: William Slacum
>
> Over night there was a mass failure of Accumulo (most likely due to too many mappers for a job). After restarting Accumulo, one of the metadata tablets failed to load. There was a log message showing a `FileNotFoundException` on the file `hdfs:///accumulo/recovery/<log id>/failed/data`. Removing the `<log id>` directory from HDFS seemed to unclog the jam and things came back (though potentially with data loss).
> I wanted to investigate why somewhere in the plumbing of `TabletServer`, `TabletServerLogger`, and `SortedLogRecovery`, an attempt was made to use the `failure` file.
> I see in `SortedLogRecovery#sort` where the marker file gets created:
> {code}
> public void sort(String name, Path srcPath, String destPath) {
> ...
>       } catch (Throwable t) {
>         try {
>           // parent dir may not exist
>           fs.mkdirs(new Path(destPath));
>           fs.create(new Path(destPath, "failed")).close();
>         } catch (IOException e) {
>           log.error("Error creating failed flag file " + name, e);
>         }
>         log.error(t, t);
>       } finally {
> ...
> {code}
> I have not stepped out to figure out where/why the `failed` files gets included in the list of recovered data dir.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)