hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Antal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13818) Extend OIV to detect FSImage corruption
Date Tue, 25 Sep 2018 13:07:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627327#comment-16627327
] 

Adam Antal commented on HDFS-13818:
-----------------------------------

Added the detailed description of the functionality to the markdown file in patch v6 - just
got through all the items from previous reviews. [~gabor.bota], if you could spare some time
and have any further suggestions please let me know.

> Extend OIV to detect FSImage corruption
> ---------------------------------------
>
>                 Key: HDFS-13818
>                 URL: https://issues.apache.org/jira/browse/HDFS-13818
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>            Reporter: Adam Antal
>            Assignee: Adam Antal
>            Priority: Major
>         Attachments: HDFS-13818.001.patch, HDFS-13818.002.patch, HDFS-13818.003.patch,
HDFS-13818.003.patch, HDFS-13818.004.patch, HDFS-13818.005.patch, HDFS-13818.006.patch, OIV_CorruptionDetector_processor.001.pdf,
OIV_CorruptionDetector_processor.002.pdf
>
>
> A follow-up Jira for HDFS-13031: an improvement of the OIV is suggested for detecting
corruptions like HDFS-13101 in an offline way.
> The reasoning is the following. Apart from a NN startup throwing the error, there is
nothing in the customer's hand that could reassure him/her that the FSImages is good or corrupted.
> Although real full checking of the FSImage is only possible by the NN, for stack traces
associated with the observed corruption cases the solution of putting up a tertiary NN is
a little bit of overkill. The OIV would be a handy choice, already having functionality
like loading the fsimage and constructing the folder structure, we just have to add the option
of detecting the null INodes. For e.g. the Delimited OIV processor can already use in disk
MetadataMap, which reduces memory consumption. Also there may be a window for parallelizing:
iterating through INodes for e.g. could be done distributed, increasing efficiency, and we
wouldn't need a high mem-high CPU setup for just checking the FSImage.
> The suggestion is to add a --detectCorruption option to the OIV which would check the
FSImage for consistency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message