hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1630) Checksum fsedits
Date Thu, 17 Feb 2011 16:22:24 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995883#comment-12995883
] 

Steve Loughran commented on HDFS-1630:
--------------------------------------

1. yes, checksums that are reasonably strong; do you need SHA1/MD5 or CRC32 enough?
2. the stream to the secondary/backup node should also be checksummed to detect problems in
machines, network cards, etc. What is complicated here is that the receiver needs the right
to say "the edit you just sent me is corrupt, resend", both ends should log this event, etc.


> Checksum fsedits
> ----------------
>
>                 Key: HDFS-1630
>                 URL: https://issues.apache.org/jira/browse/HDFS-1630
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>
> HDFS-903 calculates a MD5 checksum to a saved image, so that we could verify the integrity
of the image at the loading time.
> The other half of the story is how to verify fsedits. Similarly we could use the checksum
approach. But since a fsedit file is growing constantly, a checksum per file does not work.
I am thinking to add a checksum per transaction. Is it doable or too expensive?

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message