hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Sivachenko <trtrmi...@gmail.com>
Subject Re: distcp fails with "source and target differ in block-size"
Date Sun, 22 May 2016 17:31:00 GMT

> On 21 May 2016, at 09:34, Dmitry Sivachenko <trtrmitya@gmail.com> wrote:
> 
> 
>> On 21 May 2016, at 02:15, Chris Nauroth <cnauroth@hortonworks.com> wrote:
>> 
>> Hello Dmitry,
>> 
>> MAPREDUCE-5065 has been included in these branches for a long time.  Are
>> you certain that you passed a dfs.blocksize equal to what was used in the
>> source files?  Did all source files use the same block size?
>> 
> 
> 
> No, I am sure that I use -D dfs.blocksize=DifferentThanSourceBlockSize (I want to change
it during the copy).
> 
> I am not sure that all source files use the same block size (there are thousands of them),
but it is probably wrong to report error when I use distcp to change block size?  SInce it
is well-documented way for changing block size.
> 
> Sorry if I am missing something.
> 


So to be clear: right now with Hadoop-2.7.2 I always get "checksum mismatch" error when I
try to distcp a file with
-Ddfs.blocksize=DifferentBlockSize

And it looks like undesired behaviour, at least some stackoverflow articles suggest distcp
as a way to change blocksize of existing file:

http://stackoverflow.com/questions/29604823/change-block-size-of-existing-files-in-hadoop

So probably some time ago this did not lead to error.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message