hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Maier <wcma...@hep.wisc.edu>
Subject Re: md5sum of files on HDFS ?
Date Wed, 02 Mar 2011 18:00:01 GMT
Hi Scott-

On Wed, Mar 02, 2011 at 11:05:04AM -0600, Scott Golby wrote:
> How can I do a md5sum/sha1sum directly against files on HDFS ?

I'm not sure if you meant to exclude this option, but mounting HDFS via FUSE
would allow you to call md5sum(1) as usual. I've used this approach to migrate
several hundred TB from our old distributed storage system to HDFS. FUSE
certainly adds overhead, but it's the best way to meet your 'standard linux
command' requirement.


Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/

View raw message