hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dennis Huo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
Date Wed, 28 Mar 2018 02:59:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416732#comment-16416732

Dennis Huo commented on HDFS-13056:

Thanks for taking another look [~xiaochen]! Applied fixes in [^HDFS-13056.013.patch]. I misinterpreted
some of your previous comments (didn't notice the mention of LimitedPrivate vs Private)
which was why I removed the InterfaceStability annotations; fixed now with InterfaceStability.Unstable
in the ones with LimitedPrivate, and left the InterfaceAudience.Private ones with no InterfaceStability
annotation to let those default to Unstable.

I went ahead and removed the isDebugEnabled checks in FileChecksumHelper. In BlockChecksumHelper,
I had kept the isDebugEnabled() checks whenever the format string included calling something
like CrcUtil.toMultiCrcString which is potentially moderately expensive (not too much, but
in keeping with where other places in the code avoid "slightly" expensive things with isDebugEnabled());
prior to [this set of edits|https://github.com/apache/hadoop/pull/344/commits/391230bf2932a05da76c4a11eadea97b04de4bad#diff-c8fa735a74e1474b5d455843f850b168] I
had used wrapper "Object" that lazy-evaluates toString() to avoid isDebugEnabled() in favor
of just always passing it in as the arg to the log formatter. I guess the options are:
 # Keep the calls to CrcUtil.to*CrcString wrapped inside isDebugEnabled() as-is
 # Remove the check and just always perform that string-creation even if it won't be used
by the logger
 # Split out a "String crcDebugString = null;" and then assign it to CrcUtil.toMultiCrcString
inside an isDebugEnabled() block, and then unconditionally pass the crcDebugString into the
log format args (this was done in FileChecksumHelper just because it was convenient for sharing
a debug string between MD5 and CRC codepaths).

I have no strong preference on any of those options.

Added global timeouts of 10 seconds to TestCrcUtil and TestCrcComposer.

> Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
> ------------------------------------------------------------------------------------------------
>                 Key: HDFS-13056
>                 URL: https://issues.apache.org/jira/browse/HDFS-13056
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode, distcp, erasure-coding, federation, hdfs
>    Affects Versions: 3.0.0
>            Reporter: Dennis Huo
>            Assignee: Dennis Huo
>            Priority: Major
>         Attachments: HDFS-13056-branch-2.8.001.patch, HDFS-13056-branch-2.8.002.patch,
HDFS-13056-branch-2.8.003.patch, HDFS-13056-branch-2.8.004.patch, HDFS-13056-branch-2.8.005.patch,
HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch, HDFS-13056.002.patch, HDFS-13056.003.patch,
HDFS-13056.003.patch, HDFS-13056.004.patch, HDFS-13056.005.patch, HDFS-13056.006.patch, HDFS-13056.007.patch,
HDFS-13056.008.patch, HDFS-13056.009.patch, HDFS-13056.010.patch, HDFS-13056.011.patch, HDFS-13056.012.patch,
HDFS-13056.013.patch, Reference_only_zhen_PPOC_hadoop2.6.X.diff, hdfs-file-composite-crc32-v1.pdf,
hdfs-file-composite-crc32-v2.pdf, hdfs-file-composite-crc32-v3.pdf
> FileChecksum was first introduced in [https://issues-test.apache.org/jira/browse/HADOOP-3981] and
ever since then has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are
already stored as part of datanode metadata, and the MD5 approach is used to compute an aggregate
value in a distributed manner, with individual datanodes computing the MD5-of-CRCs per-block
in parallel, and the HDFS client computing the second-level MD5.
> A shortcoming of this approach which is often brought up is the fact that this FileChecksum
is sensitive to the internal block-size and chunk-size configuration, and thus different
HDFS files with different block/chunk settings cannot be compared. More commonly, one might
have different HDFS clusters which use different block sizes, in which case any data migration
won't be able to use the FileChecksum for distcp's rsync functionality or for verifying end-to-end
data integrity (on top of low-level data integrity checks applied at data transfer time).
> This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 during the
addition of checksum support for striped erasure-coded files; while there was some discussion
of using CRC composability, it still ultimately settled on hierarchical MD5 approach, which also adds
the problem that checksums of basic replicated files are not comparable to striped files.
> This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses CRC composition
to remain completely chunk/block agnostic, and allows comparison between striped vs replicated
files, between different HDFS instances, and possible even between HDFS and other external
storage systems. This feature can also be added in-place to be compatible with existing block
metadata, and doesn't need to change the normal path of chunk verification, so is minimally
invasive. This also means even large preexisting HDFS deployments could adopt this feature
to retroactively sync data. A detailed design document can be found here: https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message