Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
MIME-Version: 1.0
In-Reply-To: <56DEFDCB.109@ccri.com>
References: <56DEFDCB.109@ccri.com>
Date: Thu, 17 Mar 2016 20:18:27 -0400
Message-ID: 
 <CAMJ0=qF6dj9oQnXJ4qHjXhPHSr1fb1mWK+EiyHkAs4LrrkgTkQ@mail.gmail.com>
Subject: Re: Recovery file versus directory
From: Michael Wall <mjwall@gmail.com>
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary=001a114edd4aa2415d052e47af5a

--001a114edd4aa2415d052e47af5a
Content-Type: text/plain; charset=UTF-8

Andrew,

Sounds a lot like https://issues.apache.org/jira/browse/ACCUMULO-4157.
I'll look to see if what you describe could also happen with this bug.  If
you still have the gc logs, can you look for a message like "Removing WAL
for offline server" with the uuid?

Mike

On Tue, Mar 8, 2016 at 11:28 AM, Andrew Hulbert <ahulbert@ccri.com> wrote:

> Hi folks,
>
> We experienced a problem this morning with a recovery on 1.6.1 that went
> something like this:
>
> FileNotFoundException: File does not exist:
> hdfs:///accumulo/recovery/<uuid>/failed/data
>
> at Tablet.java:1410
> at Tablet.java:1233
> etc.
> at TabletServer:2923
>
> Interestingly enough, at hdfs:///accumulo/recovery/<uuid>/failed was a 0
> byte file, not a directory...and it was preventing tablets from getting
> assigned (I am not sure what caused the original failure, but I believe
> what happened is a tserver node was going down...the master indicated it
> was trying to shutdown the a tserver which was so bad off someone just
> rekicked the node).
>
> I looked through the fixes for 1.6.2,3,4,5 but didn't see anything related
> on the release notes pages but I haven't gone through all the tickets yet.
> I haven't been able to get anyone to upgrade to 1.6.5 yet and perhaps its
> already fixed.
>
> Just wondering if that's something that has been seen before?
>
> In order to fix it I just deleted the failed file and it proceeded
>
> Thanks!
>
> Andrew
>

--001a114edd4aa2415d052e47af5a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Andrew,<div><br></div><div>Sounds a lot like=C2=A0<a href=
=3D"https://issues.apache.org/jira/browse/ACCUMULO-4157">https://issues.apa=
che.org/jira/browse/ACCUMULO-4157</a>.=C2=A0 I&#39;ll look to see if what y=
ou describe could also happen with this bug.=C2=A0 If you still have the gc=
 logs, can you look for a message like &quot;Removing WAL for offline serve=
r&quot; with the uuid?</div><div><br></div><div>Mike</div></div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Mar 8, 2016 at 11:2=
8 AM, Andrew Hulbert <span dir=3D"ltr">&lt;<a href=3D"mailto:ahulbert@ccri.=
com" target=3D"_blank">ahulbert@ccri.com</a>&gt;</span> wrote:<br><blockquo=
te class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc so=
lid;padding-left:1ex">Hi folks,<br>
<br>
We experienced a problem this morning with a recovery on 1.6.1 that went so=
mething like this:<br>
<br>
FileNotFoundException: File does not exist: hdfs:///accumulo/recovery/&lt;u=
uid&gt;/failed/data<br>
<br>
at Tablet.java:1410<br>
at Tablet.java:1233<br>
etc.<br>
at TabletServer:2923<br>
<br>
Interestingly enough, at hdfs:///accumulo/recovery/&lt;uuid&gt;/failed was =
a 0 byte file, not a directory...and it was preventing tablets from getting=
 assigned (I am not sure what caused the original failure, but I believe wh=
at happened is a tserver node was going down...the master indicated it was =
trying to shutdown the a tserver which was so bad off someone just rekicked=
 the node).<br>
<br>
I looked through the fixes for 1.6.2,3,4,5 but didn&#39;t see anything rela=
ted on the release notes pages but I haven&#39;t gone through all the ticke=
ts yet. I haven&#39;t been able to get anyone to upgrade to 1.6.5 yet and p=
erhaps its already fixed.<br>
<br>
Just wondering if that&#39;s something that has been seen before?<br>
<br>
In order to fix it I just deleted the failed file and it proceeded<br>
<br>
Thanks!<span class=3D"HOEnZb"><font color=3D"#888888"><br>
<br>
Andrew<br>
</font></span></blockquote></div><br></div>

--001a114edd4aa2415d052e47af5a--