accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher <ctubb...@apache.org>
Subject Re: Question on missing RFiles
Date Fri, 11 May 2018 22:16:41 GMT
Oh, it occurs to me that this may be related to the WAL bugs that Keith
fixed for 1.9.1... which could affect the metadata table recovery after a
failure.

On Fri, May 11, 2018 at 6:11 PM Michael Wall <mjwall@gmail.com> wrote:

> Adam,
>
> Do you have GC logs?  Can you see if those missing RFiles were removed by
> the GC process?  That could indicate you somehow got old metadata info
> replayed.  Also, the rfiles increment so compare the current rfile names in
> the srv.dir directory vs what is in the metadata table.  Are the existing
> files after files in the metadata.  Finally, pick a few of the missing
> files and grep all your master and tserver logs to see if you can learn
> anything.  This sounds ungood.
>
> Mike
>
> On Fri, May 11, 2018 at 6:06 PM Christopher <ctubbsii@apache.org> wrote:
>
>> This is strange. I've only ever seen this when HDFS has reported
>> problems, such as missing blocks, or another obvious failure. What is your
>> durability settings (were WALs turned on)?
>>
>> On Fri, May 11, 2018 at 12:45 PM Adam J. Shook <adamjshook@gmail.com>
>> wrote:
>>
>>> Hello all,
>>>
>>> On one of our clusters, there are a good number of missing RFiles from
>>> HDFS, however HDFS is not/has not reported any missing blocks.  We were
>>> experiencing issues with HDFS; some flapping DataNode processes that needed
>>> more heap.
>>>
>>> I don't anticipate I can do much besides create a bunch of empty RFiles
>>> (open to suggestions).  My question is, Is it possible that Accumulo could
>>> have written the metadata for these RFiles but failed to write it to HDFS?
>>> In which case it would have been re-tried later and the data was persisted
>>> to a different RFile?  Or is it an 'RFile is in Accumulo metadata if and
>>> only if it is in HDFS' situation?
>>>
>>> Accumulo 1.8.1 on HDFS 2.6.0.
>>>
>>> Thank you,
>>> --Adam
>>>
>>

Mime
View raw message