Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Subject: Re: DistCp CRC failure modes
References: 
 <CAC3gpCY6s=eU_NEQO7yF-WF9U890PdezTrLeEDYxm6B-tDf_mQ@mail.gmail.com>
From: Akira AJISAKA <ajisakaa@oss.nttdata.co.jp>
To: user@hadoop.apache.org,
 "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>
Message-ID: <5720E832.3030609@oss.nttdata.co.jp>
Date: Thu, 28 Apr 2016 01:26:26 +0900
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.7.2
MIME-Version: 1.0
In-Reply-To: 
 <CAC3gpCY6s=eU_NEQO7yF-WF9U890PdezTrLeEDYxm6B-tDf_mQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

(Added hdfs-dev ML)

Thanks Elliot for reporting this issue.

I'm thinking this is not by design, so we should fix it.
Would you file a JIRA for this issue?
https://issues.apache.org/jira/browse/HDFS/

If you don't have time to do so, I'll file it on behalf of you.

Regards,
Akira

On 4/27/16 22:43, Elliot West wrote:
> Hello,
>
> We are using DistCp V2 to replicate data between two HDFS file systems.
> We were working on the assumption that we could rely on CRC checks to
> ensure that the data was replicated correctly. However, after examining
> the DistCp source code it seems that there are edge cases where the CRCs
> could differ and yet the copy succeeds even when we are not skipping CRC
> checks.
>
> I'm wondering whether this is by design and if so, the reasoning behind
> it? If this is a bug, I'd like to raise an issue to fix it. If it is by
> design, I'd like to propose the introduction an option for stricter CRC
> checks.
>
> The code in question is contained in the method:
>
>     org.apache.hadoop.tools.util.DistCpUtils#checksumsAreEqual(...)
>
> which can be seen here:
>
>     https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java#L457
>
>
> Specifically this code block suggests that if there is a failure when
> trying to read the source or target checksum then the method will return
> 'true', implying that the check succeeded. In actual fact we just failed
> to obtain the checksum and could perform no check.
>
>      try {
>        sourceChecksum = sourceChecksum != null ? sourceChecksum : sourceFS
>            .getFileChecksum(source);
>        targetChecksum = targetFS.getFileChecksum(target);
>      } catch (IOException e) {
>        LOG.error("Unable to retrieve checksum for " + source + " or " +
> target, e);
>      }
>      return (sourceChecksum == null || targetChecksum == null ||
>              sourceChecksum.equals(targetChecksum));
>
> Ideally I'd like to be able to configure a check where we require that
> both the source and target CRCs are retrieved and compared, and if for
> any reason either of the CRCs retrievals fail then an exception is
> thrown. I do appreciate that some FileSystems cannot return CRCs but
> these could still be handled correctly as they would simply return null
> and not throw an exception (I assume).
>
> I'd appreciate any thoughts on this matter.
>
> Elliot.


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org