hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Virajith Jalaparti (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11681) DatanodeStorageInfo#getBlockIterator() should return an iterator to an unmodifiable set.
Date Wed, 19 Apr 2017 18:21:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975221#comment-15975221
] 

Virajith Jalaparti commented on HDFS-11681:
-------------------------------------------

Currently, {{BlockManager#removeBlocksAssociatedTo(DatanodeDescriptor)}} and {{BlockManager#removeBlocksAssociatedTo(DatanodeStorageInfo)}}
work as follows: (a) get an {{Iterator<BlockInfo>}} from {{DatanodeStorageInfo#getBlockIterator()}},
(b) call {{remove}} on the iterator, and (c) finally calling {{BlockManager#removeStoredBlock}}
which eventually calls {{DatanodeStorageInfo#removeBlock()}}. 

The {{Iterator}} returned in (a) is an iterator for {{DatanodeStorageInfo.blocks}}. Calling
{{Iterator#remove}} in step (b) makes the call {{blocks.remove(b)}} in {{DatanodeStorageInfo#removeBlock()}}
redundant (a no-op). I think blocks should be removed from {{DatanodeStorageInfo.blocks}}
in {{DatanodeStorageInfo#removeBlock()}} and not by calling {{remove}} in step (b) above.

To address this issue, and to prevent unintended removal of blocks from {{DatanodeStorageInfo}}
(by calling {{remove}} on {{DatanodeStorageInfo#getBlockIterator()}}), {{DatanodeStorageInfo#getBlockIterator()}}
should be modified to return an iterator to an unmodifiable set. This would also help HDFS-9806
where {{PROVIDED}} blocks will not be removed when a single datanode fails (resulting in a
call to {{BlockManager#removeBlocksAssociatedTo(DatanodeDescriptor)}}) but all datanodes with
{{PROVIDED}} volumes fail -- so {{DatanodeStorageInfo}} should get to decide when blocks are
removed. 




> DatanodeStorageInfo#getBlockIterator() should return an iterator to an unmodifiable set.
> ----------------------------------------------------------------------------------------
>
>                 Key: HDFS-11681
>                 URL: https://issues.apache.org/jira/browse/HDFS-11681
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Virajith Jalaparti
>
> The iterator from {{DatanodeStorageInfo#getBlockIterator()}} should not be modifiable.
Otherwise, calling {{remove}} on the iterator will remove blocks from {{DatanodeStorageInfo.blocks}},
a function that has to be performed by calling {{DatanodeStorageInfo#removeBlock}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message