hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9173) Erasure Coding: Lease recovery for striped file
Date Mon, 14 Dec 2015 21:54:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056780#comment-15056780

Zhe Zhang commented on HDFS-9173:

Thanks Jing for the rebase. The patch LGTM overall. I have a few comments and some suggestions
for future work:

# The below should be removed:
  public Daemon recoverBlocks(String who, Collection<RecoveringBlock> blocks) {
    return blockRecoveryWorker.recoverBlocks(who, blocks);
# Since we are trimming null elements in {{DatanodeManager#getDatanodeStorageInfos}}, should
change the below in {{BlockUnderConstructionFeature#setExpectedLocations}} to an assertion.
    for (DatanodeStorageInfo target : targets) {
      if (target != null) {
# {{FSNamesystem#commitBlockSynchronization}} also uses {{getDatanodeStorageInfos}}. The behavior
is a little tricky. If some elements in {{newtargets}} is {{EMPTY_DATANODE_ID}}, or not found
in DNManager, then {{dsInfos}} will be shorter than a full stripe. But we are always assigning
block IDs based on an element's offset in {{dsInfos}}.
                if (storedBlock.isStriped()) {
                  bi.setBlockId(bi.getBlockId() + i);
# Should update to "If some internal blocks reach the safe length"
      // If some internal blocks are longer than safe length, convert them to
      // to RUR replicas.
# {{newLocs}} and {{newStorages}} should be constructed in the same loop as {{rurList}}. Something
like below. To avoid the double loop.
final DatanodeID[] newLocs = new DatanodeID[totalBlkNum];
Arrays.fill(newLocs, EMPTY_DATANODE_ID);
for (BlockRecord r : syncList) {
	if (r.rInfo.getNumBytes() >= newSize) {
		newLocs[blockIndex] = r.id;
		newStorages[blockIndex] = r.storageID;

> Erasure Coding: Lease recovery for striped file
> -----------------------------------------------
>                 Key: HDFS-9173
>                 URL: https://issues.apache.org/jira/browse/HDFS-9173
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>            Assignee: Walter Su
>         Attachments: HDFS-9173.00.wip.patch, HDFS-9173.01.patch, HDFS-9173.02.step125.patch,
HDFS-9173.03.patch, HDFS-9173.04.patch, HDFS-9173.05.patch, HDFS-9173.06.patch, HDFS-9173.07.patch

This message was sent by Atlassian JIRA

View raw message