hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15273) distcp to downgrade on checksum algorithm mismatch to "files unchanged"
Date Wed, 07 Mar 2018 14:37:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389633#comment-16389633

Steve Loughran commented on HADOOP-15273:

oh, and the text is actually wrong, as the arg is "-skipcrccheck"

> distcp to downgrade on checksum algorithm mismatch to "files unchanged"
> -----------------------------------------------------------------------
>                 Key: HADOOP-15273
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15273
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: tools/distcp
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Priority: Minor
> When using distcp without {{-skipCRC}} . If there's a checksum mismatch between src and
dest store types (e.g hdfs to s3), then the error message will talk about blocksize, even
when its the underlying checksum protocol itself which is the cause for failure
> bq. Source and target differ in block-size. Use -pb to preserve block-sizes during copy.
Alternatively, skip checksum-checks altogether, using -skipCrc. (NOTE: By skipping checksums,
one runs the risk of masking data-corruption during file-transfer.)
> IF the checksum types are fundamentally different, the error message should say so

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message