hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Aquilina <jaquil...@eagleeyet.net>
Subject Re: Comparing CheckSum of Local and HDFS File
Date Sun, 16 Aug 2015 14:44:45 GMT
 

Correct me if I am wrong but the command you ran on the cluster seems to
be doing a CRC check as well. I am still a novice to hadoop but that is
the most obvious thing i see in the output below. 

---
Regards,
Jonathan Aquilina
Founder Eagle Eye T

On 2015-08-07 12:34, Shashi Vishwakarma wrote: 

> Hi 
> 
> I have a small confusion regarding checksum verification.Lets say , i have a file abc.txt
and I transferred this file to hdfs. How do I ensure about data integrity? 
> 
> I followed below steps to check that file is correctly transferred. 
> 
> ON LOCAL FILE SYSTEM: 
> 
> md5sum abc.txt 
> 
> 276fb620d097728ba1983928935d6121 TestFile 
> 
> ON HADOOP CLUSTER : 
> 
> hadoop fs -checksum /abc.txt 
> 
> /abc.txt MD5-of-0MD5-of-512CRC32C 000002000000000000000000911156a9cf0d906c56db7c8141320df0

> 
> Both output looks different to me. Let me know if I am doing anything wrong. 
> 
> How do I verify if my file is transferred properly into HDFS? 
> 
> Thanks 
> Shashi
 
Mime
View raw message