hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2476) More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks
Date Fri, 01 Feb 2013 14:48:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568767#comment-13568767

Kihwal Lee commented on HDFS-2476:

Committed to the current branch-0.23.
> More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks
> ----------------------------------------------------------------------------------------
>                 Key: HDFS-2476
>                 URL: https://issues.apache.org/jira/browse/HDFS-2476
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 0.23.0
>            Reporter: Tomasz Nykiel
>            Assignee: Tomasz Nykiel
>             Fix For: 2.0.0-alpha
>         Attachments: hashStructures.patch, hashStructures.patch-2, hashStructures.patch-3,
hashStructures.patch-4, hashStructures.patch-5, hashStructures.patch-6, hashStructures.patch-7,
hashStructures.patch-8, hashStructures.patch-9
> This patch introduces two hash data structures for storing under-replicated, over-replicated
and invalidated blocks. 
> 1. LightWeightHashSet
> 2. LightWeightLinkedSet
> Currently in all these cases we are using java.util.TreeSet which adds unnecessary overhead.
> The main bottlenecks addressed by this patch are:
> -cluster instability times, when these queues (especially under-replicated) tend to grow
quite drastically,
> -initial cluster startup, when the queues are initialized, after leaving safemode,
> -block reports,
> -explicit acks for block addition and deletion
> 1. The introduced structures are CPU-optimized.
> 2. They shrink and expand according to current capacity.
> 3. Add/contains/delete ops are performed in O(1) time (unlike current log n for TreeSet).
> 4. The sets are equipped with fast access methods for polling a number of elements (get+remove),
which are used for handling the queues.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message