accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher <>
Subject Re: Question on missing RFiles
Date Fri, 11 May 2018 22:16:41 GMT
Oh, it occurs to me that this may be related to the WAL bugs that Keith
fixed for 1.9.1... which could affect the metadata table recovery after a

On Fri, May 11, 2018 at 6:11 PM Michael Wall <> wrote:

> Adam,
> Do you have GC logs?  Can you see if those missing RFiles were removed by
> the GC process?  That could indicate you somehow got old metadata info
> replayed.  Also, the rfiles increment so compare the current rfile names in
> the srv.dir directory vs what is in the metadata table.  Are the existing
> files after files in the metadata.  Finally, pick a few of the missing
> files and grep all your master and tserver logs to see if you can learn
> anything.  This sounds ungood.
> Mike
> On Fri, May 11, 2018 at 6:06 PM Christopher <> wrote:
>> This is strange. I've only ever seen this when HDFS has reported
>> problems, such as missing blocks, or another obvious failure. What is your
>> durability settings (were WALs turned on)?
>> On Fri, May 11, 2018 at 12:45 PM Adam J. Shook <>
>> wrote:
>>> Hello all,
>>> On one of our clusters, there are a good number of missing RFiles from
>>> HDFS, however HDFS is not/has not reported any missing blocks.  We were
>>> experiencing issues with HDFS; some flapping DataNode processes that needed
>>> more heap.
>>> I don't anticipate I can do much besides create a bunch of empty RFiles
>>> (open to suggestions).  My question is, Is it possible that Accumulo could
>>> have written the metadata for these RFiles but failed to write it to HDFS?
>>> In which case it would have been re-tried later and the data was persisted
>>> to a different RFile?  Or is it an 'RFile is in Accumulo metadata if and
>>> only if it is in HDFS' situation?
>>> Accumulo 1.8.1 on HDFS 2.6.0.
>>> Thank you,
>>> --Adam

View raw message