hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3859) QJM: implement md5sum verification
Date Tue, 28 Aug 2012 20:17:07 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443483#comment-13443483
] 

Todd Lipcon commented on HDFS-3859:
-----------------------------------

Sure, it's overkill, but it's not that expensive and we already have an implementation of
it sitting around. It's also handy because "md5sum" is commonly available on the command line,
and we use it for FSImages already as well. Performance-wise, my laptop can md5sum at about
500MB/sec, so given that log segments under recovery are likely to be much smaller than 500M,
I don't think we should be concerned about that.
                
> QJM: implement md5sum verification
> ----------------------------------
>
>                 Key: HDFS-3859
>                 URL: https://issues.apache.org/jira/browse/HDFS-3859
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: QuorumJournalManager (HDFS-3077)
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>
> When the QJM passes journal segments between nodes, it should use an md5sum field to
make sure the data doesn't get corrupted during transit. This also serves as an extra safe-guard
to make sure that the data is consistent across all nodes when finalizing a segment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message