hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Korb, Michael [USA]" <Korb_Mich...@bah.com>
Subject HDFS Data Integrity in copyToLocal
Date Wed, 18 Dec 2013 05:43:24 GMT

How can I verify the integrity of files copied to local from HDFS? Does HDFS store MD5s of
full files anywhere? From what I can find, FileSystem.getFileChecksum() is relevant to replication
and not comparison across filesystems (http://stackoverflow.com/questions/14563245/hdfs-file-checksum).

The Data Integrity section in HDFS Architecture (http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html)
does not make it clear if, or how, copyToLocal verifies the integrity of the copied file.


View raw message