Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
MIME-Version: 1.0
In-Reply-To: <3a3d276b-bd3c-61bf-58f6-5c885c253f9c@ccri.com>
References: <CA+Vtps4z6=iRQfhDTA7TbpEF1cpbc2guzzxxVk-TjFyRxG4GEw@mail.gmail.com>
 <CAMJ0=qH70bKgjxwOceLBMJbSRRouoB7Do96Yn31kU1zunExqtA@mail.gmail.com>
 <CA+Vtps4jFW2VVeinbhWLsvy4Lt9nTjjUiCvzrmo9UA8hSduesA@mail.gmail.com> <3a3d276b-bd3c-61bf-58f6-5c885c253f9c@ccri.com>
From: Michael Wall <mjwall@gmail.com>
Date: Tue, 18 Oct 2016 09:46:08 -0400
Message-ID: <CAMJ0=qHDhhxsrLYyXmCHA3eGoEX9JAQ0dUD=Xjgcm8m4_EWnGA@mail.gmail.com>
Subject: Re: java.IO.EOFException: ..../accumulo/recovery/.../part-r-00000/index
 not a SequenceFile.
To: Accumulo User List <user@accumulo.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c19ad5c344aa5053f23eaf5
archived-at: Tue, 18 Oct 2016 13:46:16 -0000

--94eb2c19ad5c344aa5053f23eaf5
Content-Type: text/plain; charset=UTF-8

Andrew,

That is what I was going to suggest you try.  Where is that "Unable to find
recovery files for extent" log?  Anyway we can see some actual logs?

Are all the WALs there?  Do you find any of the WAL deleted by GC in the gc
logs?  Do you find any duplicates WALs in the HDFS trash?

On Tue, Oct 18, 2016 at 9:32 AM, Andrew Hulbert <ahulbert@ccri.com> wrote:

> Mike,
> For one of the WALs I backed up the recovery directory and that initiated
> a new recovery attempt as indicated in the tserver debug log...
>
> Then the exception was thrown:
>
> Unable to find recovery files for extent xxxxxx logentry xxxxx
> hdfs://path/to/wal/yyyy
>
> Any ideas? I figure we can zero out the WAL and it will go on with life
> but it would be nice to try and get the data!
>
> Thanks!
>
>
> On 10/18/2016 08:55 AM, Jeff Kubina wrote:
>
>
> On Tue, Oct 18, 2016 at 6:32 AM, Michael Wall <mjwall@gmail.com> wrote:
>
>> Take a look at the master logs for where the WAL was sorted to the /accumulo/recovery/...
>> directory.  Then look to see if those WALs are still around and contain
>> content.
>>
>
> Checked one of them, yes it is around with content.
>
> Where is this this EOF exception, on a tserver?
>>
>
> Yes, the tserver.
>
>
>> Is the master log complaining about anything?
>>
>
> Repeating a message similar to the tserver but also that the tablet
> assignment failed for the tserver.
>
> tservers are not balancing because of all this.
>
>
>
>

--94eb2c19ad5c344aa5053f23eaf5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Andrew,<br><br></div>That is what I was going to=
 suggest you try.=C2=A0 Where is that &quot;Unable to find recovery files f=
or extent&quot; log?=C2=A0 Anyway we can see some actual logs?<br><br></div=
>Are all the WALs there?=C2=A0 Do you find any of the WAL deleted by GC in =
the gc logs?=C2=A0 Do you find any duplicates WALs in the HDFS trash?<br></=
div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Oct 1=
8, 2016 at 9:32 AM, Andrew Hulbert <span dir=3D"ltr">&lt;<a href=3D"mailto:=
ahulbert@ccri.com" target=3D"_blank">ahulbert@ccri.com</a>&gt;</span> wrote=
:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-le=
ft:1px #ccc solid;padding-left:1ex">
 =20
   =20
 =20
  <div bgcolor=3D"#FFFFFF" text=3D"#000000">
    <p>Mike, <br>
    </p>
    For one of the WALs I backed up the recovery directory and that
    initiated a new recovery attempt as indicated in the tserver debug
    log...<br>
    <br>
    Then the exception was thrown:<br>
    <br>
    Unable to find recovery files for extent xxxxxx logentry xxxxx
    hdfs://path/to/wal/yyyy<br>
    <br>
    Any ideas? I figure we can zero out the WAL and it will go on with
    life but it would be nice to try and get the data!<br>
    <br>
    Thanks!<div><div class=3D"h5"><br>
    <br>
    <div class=3D"m_-9073803852204155430moz-cite-prefix">On 10/18/2016 08:5=
5 AM, Jeff Kubina
      wrote:<br>
    </div>
    <blockquote type=3D"cite">
      <div dir=3D"ltr">
        <div class=3D"gmail_extra"><br>
          <div class=3D"gmail_quote">On Tue, Oct 18, 2016 at 6:32 AM,
            Michael Wall <span dir=3D"ltr">&lt;<a href=3D"mailto:mjwall@gma=
il.com" target=3D"_blank">mjwall@gmail.com</a>&gt;</span>
            wrote:<br>
            <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo=
rder-left:1px #ccc solid;padding-left:1ex">
              <div>Take a look at the master logs for where the WAL was
                sorted to the=C2=A0<span style=3D"font-size:12.8px">/accumu=
lo/recovery/...
                  directory.=C2=A0 Then look to see if those WALs are still
                  around and contain content.</span></div>
            </blockquote>
            <div><br>
            </div>
            <div>Checked one of them, yes it is around with content.<br>
              <br>
            </div>
            <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo=
rder-left:1px #ccc solid;padding-left:1ex">
              <div>Where is this this EOF exception, on a tserver?</div>
            </blockquote>
            <div><br>
            </div>
            <div>Yes, the tserver.<br>
              =C2=A0<br>
            </div>
            <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo=
rder-left:1px #ccc solid;padding-left:1ex">
              <div>Is the master log complaining about anything?</div>
            </blockquote>
            <div><br>
            </div>
            <div>Repeating a message similar to the tserver but also
              that the tablet assignment failed for the tserver.<br>
              <br>
            </div>
            <div>tservers are not balancing because of all this.<br>
            </div>
          </div>
          <br>
          <div class=3D"m_-9073803852204155430gmail_signature" data-smartma=
il=3D"gmail_signature">
            <div dir=3D"ltr">
              <div><br>
              </div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
  </div></div></div>

</blockquote></div><br></div>

--94eb2c19ad5c344aa5053f23eaf5--