hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongjun Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11187) Optimize disk access for last partial chunk checksum of Finalized replica
Date Sat, 06 Jan 2018 01:02:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314247#comment-16314247

Yongjun Zhang commented on HDFS-11187:


Thanks for investigating the issue and patch. The changes make sense to me, some comments:

1. {{calculateLastPartialChunkChecksum()}} can be renamed to {{loadLastPartialChunkChecksum}},
and it can be changed to "void" instead of returning {{byte[]}}, because it calls {{setLastPartialChunkChecksum(lastChecksum)}}

2. {{loadPartialChunkChecksumForFinalized}} can be renamed to {{getPartialChunkChecksumForFinalized}},
because we only do the load when needed inside the implementation

3. Because of 1, {{loadPartialChunkChecksumForFinalized}}  can be streamlined to have a shared
return statement. Currently we load checksum when 
   if (replicaVisibleLength % CHUNK_SIZE != 0 &&
        finalized.getLastPartialChunkChecksum() == null) {
that means we don't load checksum when replicaVisibleLength is CHUNK_SIZE, in which case it's
possible checksum is not loaded (thus null), suggest to add a comment how that case is handled.

4. Move the comment
       // The number of bytes per checksum here determines the alignment
        // of reads: we always start reading at a checksum chunk boundary,
        // even if the checksum type is NULL. So, choosing too big of a value
        // would risk sending too much unnecessary data. 512 (1 disk sector)
        // is likely to result in minimal extra IO.
to where CHUNK_SIZE is defined.

5. There is a possible race condition that a finalized replica is moved to RBW then modified
by another writer, and the reader is trying to access the same replica. We can discuss this
situation separately.


> Optimize disk access for last partial chunk checksum of Finalized replica
> -------------------------------------------------------------------------
>                 Key: HDFS-11187
>                 URL: https://issues.apache.org/jira/browse/HDFS-11187
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>         Attachments: HDFS-11187.001.patch, HDFS-11187.002.patch
> The patch at HDFS-11160 ensures BlockSender reads the correct version of metafile when
there are concurrent writers.
> However, the implementation is not optimal, because it must always read the last partial
chunk checksum from disk while holding FsDatasetImpl lock for every reader. It is possible
to optimize this by keeping an up-to-date version of last partial checksum in-memory and reduce
disk access.
> I am separating the optimization into a new jira, because maintaining the state of in-memory
checksum requires a lot more work.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message