hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: distcp fails with "source and target differ in block-size"
Date Tue, 24 May 2016 17:03:27 GMT
There is also some discussion on that JIRA considering a checksum strategy
independent of block size.  I don't think anything was ever implemented
though, and there would be some drawbacks to that approach.  Sorry if this
caused confusion.

--Chris Nauroth

On 5/24/16, 9:55 AM, "Dmitry Sivachenko" <trtrmitya@gmail.com> wrote:

>> On 24 May 2016, at 19:53, Chris Nauroth <cnauroth@hortonworks.com>
>> Hello Dmitry,
>> To clarify, the intent of MAPREDUCE-5065 was to message the user that
>> using different block sizes on source and destination might cause a
>> failure to checksum mismatch.  The message to the user recommends either
>> the -pb (preserve block size) or -skipCrc (skip checksum validation) as
>> potential workarounds.  The intent of that patch was not to silently
>> proceed and report success when the block sizes are different, although
>> there was some discussion of that on the issue as a proposed solution.
>> To the best of my knowledge, this behavior hasn't really changed.  Only
>> the messaging to the user has changed to advise on some potential
>> workarounds.
>Okay, sorry for misunderstanding, I thought the intention was to make
>checksum blocksize-independent (which would be very intuitive).
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org

To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org

View raw message