hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rodrigo Schmidt (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1111) getCorruptFiles() should give some hint that the list is not complete
Date Wed, 23 Jun 2010 01:57:52 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881512#action_12881512

Rodrigo Schmidt commented on HDFS-1111:

Thanks for the comments, Konstantin!

You are right about 1. I'll change that.

As for 2, those words make a big difference for ops people, specially if they are running
fsck -list-corruptfiles on a subdirectory. Knowing that the list is empty because there are
no corrupt files instead of maybe thinking it's empty because the list reported has more than
the limit number of corrupt files in a different directory makes all the difference for them.

Let me give you an example: some time ago we had a problem and many files got corrupted. We
were using fsck -list-corruptfiles because it was faster and direct, but we wanted to focus
on important directories first. Wee ran fsck -list-corruptfiles /path/to/important/dir but
it returned an empty list. This was weird because we knew there were corrupt files there.
The problem was that we filter the directory after we get the list reported from the namenode
and the list is limited. For that reason, it was truncated with files in different directories
and reported ambiguous output.

Although the new code makes just a minor change to the interface, its meaning makes a huge
impact to the user.

> getCorruptFiles() should give some hint that the list is not complete
> ---------------------------------------------------------------------
>                 Key: HDFS-1111
>                 URL: https://issues.apache.org/jira/browse/HDFS-1111
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>         Attachments: HADFS-1111.0.patch
> If the list of corruptfiles returned by the namenode doesn't say anything if the number
of corrupted files is larger than the call output limit (which means the list is not complete).
There should be a way to hint incompleteness to clients.
> A simple hack would be to add an extra entry to the array returned with the value null.
Clients could interpret this as a sign that there are other corrupt files in the system.
> We should also do some rephrasing of the fsck output to make it more confident when the
list is not complete and less confident when the list is known to be incomplete.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message