hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-492) Expose corrupt replica/block information
Date Thu, 30 Jul 2009 22:03:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737324#action_12737324

Suresh Srinivas commented on HDFS-492:

Comments on the previous version of the patch. I will review the next version that works with
# {{BlockManager.numBlocksWithCorruptReplicas()}} 
#* Currently this can be obtained by by looking {{BlockManager.corruptReplicaBlocksCount}}
(unsynchronized). However this count is updated every {{dfs.replication.interva}}. If that
approximate number is sufficent, then there is no need to introduce this method. Also existing
method {{FSNamesystem.getCorruptReplicaBlocks()}} instead of new method added {{FSNamesystem.numBlocksWithCorruptReplicas()}},
can be used. 
#* If the number needs to be accurate, then remove {{BlockManager.corruptReplicaBlocksCount}}
and synchronize the method {{FSNamesystem.getCorruptReplicaBlocks()}} and use {{BlockManager.numBlocksWithCorruptReplicas()}}
to get the count.
# {{CorruptReplicasMap.numBlocksWithCorruptReplicas()}} - instead of adding this, use existing
method {{CorruptReplicasMap.size()}}
# Comments in {{CorruptReplicasMap.java}} can be shorted. No need to explain how the code
works - a brief comment will be sufficient . Also remove the comment with XXXBZ
# {{CorruptReplicasMap.getCorruptReplicaBlockIds}} - it is better to take {{long}} as second
param instead of object {{Long}}. Also returning {{long[]}} is better than {{List<Long>}}
# TestCorruptReplicaInfo.java - use assertEquals() instead of assertTrue(). When the test
fails it gives information about expected Vs actual value.
# TestCorruptReplicaInfo.block_map - can just be a HashMap

# Generic - instead of {{n}}, use {{numExpectedBlocks}}
# CorruptReplicasMap.java - better to change the type of corruptReplicasMap from TreeMap to

> Expose corrupt replica/block information
> ----------------------------------------
>                 Key: HDFS-492
>                 URL: https://issues.apache.org/jira/browse/HDFS-492
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, name-node
>    Affects Versions: 0.21.0
>            Reporter: Bill Zeller
>            Assignee: Bill Zeller
>            Priority: Minor
>             Fix For: 0.21.0
>         Attachments: hdfs-492-4.patch, hdfs-492-5.patch
>   Original Estimate: 48h
>  Remaining Estimate: 48h
> This adds two additional functions to FSNamesystem to provide more information about
corrupt replicas. It also adds two servlets to the namenode that provide information (in JSON)
about all blocks with corrupt replicas as well as information about a specific block. It also
changes the file browsing servlet by adding a link from block ids to the above mentioned block
information page.
> These JSON pages are designed to be used by client side tools which wish to analyze corrupt
block/replicas. The only change to an existing (non-servlet) class is described below.  
> Currently, CorruptReplicasMap stores a map of corrupt replica information and allows
insertion and deletion. It also gives information about the corrupt replicas for a specific
block. It does not allow iteration over all corrupt blocks. Two additional functions will
be added to FSNamesystem (which will call BlockManager which will call CorruptReplicasMap).
The first will return the size of the corrupt replicas map, which represents the number of
blocks that have corrupt replicas (but less than the number of corrupt replicas if a block
has multiple corrupt replicas). The second will allow "paging" through a list of block ids
that contain corrupt replicas:
> {{public synchronized List<Long> getCorruptReplicaBlockIds(int n, Long startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the block id offset.
To prevent a large number of items being returned at one time, n is constrained to 0 <=
{{n}} <= 100. If {{startingBlockId}} is null, up to {{n}} items are returned starting at
the beginning of the list. Ordering is enforced through the internal use of TreeMap in CorruptReplicasMap.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message