hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rodrigo Schmidt (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1111) getCorruptFiles() should give some hint that the list is not complete
Date Tue, 29 Jun 2010 09:24:51 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883481#action_12883481

Rodrigo Schmidt commented on HDFS-1111:

The RaidNode is not currently using this API, although its use was one of the motivations
I had for adding getCorruptFiles() to ClientProtocol. Originally, raid was part of HDFS and
I could certainly see how Raid (and possibly other parts of HDFS) could benefit from this
as an RPC to the namenode. I thought the others saw it too because when I got to HDFS-729,
having getCorruptFiles() on ClientProtocol was not under discussion anymore.

The JIRA that is responsible for making the RaidNode call getCorruptFiles is HDFS-1171. Most
probably we will have to extend DistributedFileSystem to export getCorruptFiles(). That's
why I said we don't have to be external to HDFS, but we can be external to the namenode.

On the other hand, if we do take getCorruptFiles() out of ClientProtocol, we will make HDFS-1171
overly complicated or expensive.

I really think the correct design choice is to export basic APIs like getCorruptFiles() as
RPCs and build services like fsck and raid completely outside the namenode. After looking
at the fsck code from the inside out and having experienced how it can sometimes compromise
the whole filesystem because the namenode is using most of its resources to calculate outputs
for fsck requests, I'm convinced it should be outside the namenode. For the sake of horizontal
scalability of the namenode, we should be working in redesigning things like the current fsck
implementation, instead of reinforcing it.

That's what I meant when I said we should be taking things out of the namenode. In my opinion,
even if my case about having other parts of HDFS call getCorruptFiles() is not convincing,
taking it out of ClientProtocol only reinforces the design choice of running fsck inside the
namenode, which I think is bad. As we have more and more discussions about a distributed namenode,
things like fsck should be the first ones running externally to it (to the namenode, not to
HDFS). I see this as a low-hanging fruit towards a more scalable and distributed namenode.

> getCorruptFiles() should give some hint that the list is not complete
> ---------------------------------------------------------------------
>                 Key: HDFS-1111
>                 URL: https://issues.apache.org/jira/browse/HDFS-1111
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>         Attachments: HADFS-1111.0.patch
> If the list of corruptfiles returned by the namenode doesn't say anything if the number
of corrupted files is larger than the call output limit (which means the list is not complete).
There should be a way to hint incompleteness to clients.
> A simple hack would be to add an extra entry to the array returned with the value null.
Clients could interpret this as a sign that there are other corrupt files in the system.
> We should also do some rephrasing of the fsck output to make it more confident when the
list is not complete and less confident when the list is known to be incomplete.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message