hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9173) Erasure Coding: Lease recovery for striped file
Date Mon, 14 Dec 2015 21:54:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056780#comment-15056780
] 

Zhe Zhang commented on HDFS-9173:
---------------------------------

Thanks Jing for the rebase. The patch LGTM overall. I have a few comments and some suggestions
for future work:

# The below should be removed:
{code}
  public Daemon recoverBlocks(String who, Collection<RecoveringBlock> blocks) {
    return blockRecoveryWorker.recoverBlocks(who, blocks);
  }
{code}
# Since we are trimming null elements in {{DatanodeManager#getDatanodeStorageInfos}}, should
change the below in {{BlockUnderConstructionFeature#setExpectedLocations}} to an assertion.
{code}
    for (DatanodeStorageInfo target : targets) {
      if (target != null) {
        numLocations++;
      }
    }
{code}
# {{FSNamesystem#commitBlockSynchronization}} also uses {{getDatanodeStorageInfos}}. The behavior
is a little tricky. If some elements in {{newtargets}} is {{EMPTY_DATANODE_ID}}, or not found
in DNManager, then {{dsInfos}} will be shorter than a full stripe. But we are always assigning
block IDs based on an element's offset in {{dsInfos}}.
{code}
                if (storedBlock.isStriped()) {
                  bi.setBlockId(bi.getBlockId() + i);
                }
{code}
# Should update to "If some internal blocks reach the safe length"
{code}
      // If some internal blocks are longer than safe length, convert them to
      // to RUR replicas.
{code}
# {{newLocs}} and {{newStorages}} should be constructed in the same loop as {{rurList}}. Something
like below. To avoid the double loop.
{code}
final DatanodeID[] newLocs = new DatanodeID[totalBlkNum];
Arrays.fill(newLocs, EMPTY_DATANODE_ID);
for (BlockRecord r : syncList) {
	...
	if (r.rInfo.getNumBytes() >= newSize) {
		rurList.add(r);
		newLocs[blockIndex] = r.id;
		newStorages[blockIndex] = r.storageID;
	}
}
{code}

> Erasure Coding: Lease recovery for striped file
> -----------------------------------------------
>
>                 Key: HDFS-9173
>                 URL: https://issues.apache.org/jira/browse/HDFS-9173
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>            Assignee: Walter Su
>         Attachments: HDFS-9173.00.wip.patch, HDFS-9173.01.patch, HDFS-9173.02.step125.patch,
HDFS-9173.03.patch, HDFS-9173.04.patch, HDFS-9173.05.patch, HDFS-9173.06.patch, HDFS-9173.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message