hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Brown <tombrow...@gmail.com>
Subject Re: HDFS copyToLocal and get crc option
Date Fri, 31 Jan 2014 16:30:42 GMT
What is the right way to use the "-crc" option with hadoop dfs -copyToLocal?

Is this the wrong list?


On Tue, Jan 28, 2014 at 11:53 AM, Tom Brown <tombrown52@gmail.com> wrote:

> I am archiving a large amount of data out of my HDFS file system to a
> separate shared storage solution (There is not much HDFS space left in my
> cluster, and upgrading it is not an option right now).
> I understand that HDFS internally manages checksums and won't succeed if
> the data doesn't match the CRC, so I'm not worried about corruption when
> reading from HDFS.
> However, I want to store the HDFS crc calculations alongside the data
> files after exporting them. I thought the "hadoop dfs -copyToLocal -crc
> <hdfs-source> <local-dest>" command would work, but it always gives me the
> error "-crc option is not valid when source file system does not have crc
> files"
> Can someone explain what exactly that option does, and when (if ever) it
> should be used?
> Thanks in advance!
> --Tom

View raw message