hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Ryan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1111) getCorruptFiles() should give some hint that the list is not complete
Date Mon, 26 Apr 2010 22:11:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861139#action_12861139
] 

Andrew Ryan commented on HDFS-1111:
-----------------------------------

Overall this feature is a great thing to have. Running full fsck is not practical on a very
large cluster, it takes a long time (> 1 hour) and uses a lot of resources on the namenode.

Here's an example of the current output:
"""
hadoop fsck / -list-corruptfiles
Here are a few files that may be corrupted:
===========================================
/tmp/hive-root/hive_2010-04-23_15-01-29_568_7146600693398653716/_tmp.10001/_tmp.attempt_201004162336_104941_r_000119_0

There is at least 1 corrupt file under '/', which is CORRUPT
"""

>From an admin's point of view this message is bad for several reasons. "a few files that
may be corrupted" is not very confident language :)
1) It is not clear if this is the only file corrupted or if there are more. 
2) It is not clear if this file is or is not corrupted.
3) It is not clear if there is any recourse, other than running 'fsck /', to get more details.

Thanks!

> getCorruptFiles() should give some hint that the list is not complete
> ---------------------------------------------------------------------
>
>                 Key: HDFS-1111
>                 URL: https://issues.apache.org/jira/browse/HDFS-1111
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>
> If the list of corruptfiles returned by the namenode doesn't say anything if the number
of corrupted files is larger than the call output limit (which means the list is not complete).
There should be a way to hint incompleteness to clients.
> A simple hack would be to add an extra entry to the array returned with the value null.
Clients could interpret this as a sign that there are other corrupt files in the system.
> We should also do some rephrasing of the fsck output to make it more confident when the
list is not complete and less confident when the list is known to be incomplete.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message