hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11634) Optimize BlockIterator when interating starts in the middle.
Date Fri, 14 Apr 2017 21:53:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969582#comment-15969582
] 

Konstantin Shvachko commented on HDFS-11634:
--------------------------------------------

Hey [~rushabh.shah], what I mean is that {{Iterator}} interface does not have API to iterate
backward or reset the iterator to a random member of a collections.
The snippet you are referring to indeed loops back to the beginning, but that should be pretty
rare, when the initial random starting point was chosen too close to the end of the list.
I don't think we need to optimize the circular jump. Do you?

> Optimize BlockIterator when interating starts in the middle.
> ------------------------------------------------------------
>
>                 Key: HDFS-11634
>                 URL: https://issues.apache.org/jira/browse/HDFS-11634
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.6.5
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: HDFS-11634.001.patch, HDFS-11634.002.patch, HDFS-11634.003.patch,
HDFS-11634.004.patch, HDFS-11643.005.patch
>
>
> {{BlockManager.getBlocksWithLocations()}} needs to iterate blocks from a randomly selected
{{startBlock}} index. It creates an iterator which points to the first block and then skips
all blocks until {{startBlock}}. It is inefficient when DN has multiple storages. Instead
of skipping blocks one by one we can skip entire storages. Should be more efficient on average.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message