hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rodrigo Schmidt (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1111) getCorruptFiles() should give some hint that the list is not complete
Date Fri, 25 Jun 2010 22:51:51 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882746#action_12882746

Rodrigo Schmidt commented on HDFS-1111:

If I remember correctly, there is another message when the list is empty telling that no corrupt
files could have been found. That part of the code was not changed by this patch.

The -list-corruptfiles option on fsck emerged after a number of incremental changes, in the
very organic way open source works.

The reason why getCorruptFiles() belongs to ClientProtocol is to allow external services to
query it. We are working on having the RaidNode use this call to automatically recover corrupted
blocks. I guess this shows we did think some things through.

On a design related note, if we want to build a truly distributed namenode, we should be thinking
about taking things out of it, like fsck itself.

Last, HDFS-1265 seems to be an orthogonal issue, since it doesn't guarantee the list is complete.
Adding filters DOES NOT prevent the queried directory from having thousands of corrupted files,
out of which, only a few hundred will be reported.

> getCorruptFiles() should give some hint that the list is not complete
> ---------------------------------------------------------------------
>                 Key: HDFS-1111
>                 URL: https://issues.apache.org/jira/browse/HDFS-1111
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>         Attachments: HADFS-1111.0.patch
> If the list of corruptfiles returned by the namenode doesn't say anything if the number
of corrupted files is larger than the call output limit (which means the list is not complete).
There should be a way to hint incompleteness to clients.
> A simple hack would be to add an extra entry to the array returned with the value null.
Clients could interpret this as a sign that there are other corrupt files in the system.
> We should also do some rephrasing of the fsck output to make it more confident when the
list is not complete and less confident when the list is known to be incomplete.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message