hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manoj Govindassamy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-10999) Introduce separate stats for Replicated and Erasure Coded Blocks apart from the current Aggregated stats
Date Fri, 07 Apr 2017 00:57:41 GMT

     [ https://issues.apache.org/jira/browse/HDFS-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Manoj Govindassamy updated HDFS-10999:
    Attachment: HDFS-10999.01.patch

Based on previous discussions, attaching v01 patch to address the following.
* {{ClientProtocol}} extended to support {{ReplicatedBlocksStats getReplicatedBlocksStats()}}
and {{ECBlockGroupsStats getECBlockGroupsStats()}} apart from the current {{long[] getStats()}}
* Introduced new MBeans {{ECBlockGroupsStatsMBean}} and {{ReplicatedBlocksStatsMBean}} for
the consumers like {{DFSAdmin}}, WebUI.
* {{FSNamesystemMBean}} will continue to carry aggregated stats combining both replicates
and ec block stats. Since these are aggregated stats now, deprecated the methods to use proper
* {{FSNamesystem}} now implements {{ECBlockGroupsStatsMBean}} and {{ReplicatedBlocksStatsMBean}}
apart from the already implemented {{FSNamesystemMBean}}
* {{BlockMAnager}} changes to expose the stats specific to Replicated and EC Blocks
* {{LowRedundancyBlocks}}, {{CorruptReplicasMap}} and {{InvalidateBlocks}} updated to track
Replicated and EC Blocks separately using LongAccumulators. Already existing aggregate blocks
tracking size() methods are not altered for backward compatibility.
* {{PBHelperClient}}, {{ClientNamenodeProtocolTranslatorPB}}, {{ClientNamenodeProtocolServerSideTranslatorPB}}
are updated to plumb in the new ClientProtocol services.
* {{ClientNameNodeProtocol.proto}} updated to define the new ClientProtocol services proto
buf messages.
* {{TestNameNodeMetrics}}, {{TestUnderReplicatedBlocks}} are updated to verify the new stats.
Several other tests are updated to verify the needed block counts.
* PS: {{DfsAdmin -report}} and WebUI are not updated to make use of the newer infrastructure.
Probably after we finalize on this infra, I can take up the consumers separately in a new

[~andrew.wang], [~tasanuma0829], [~jojochuang], can you please take a look at the attached
patch ?

> Introduce separate stats for Replicated and Erasure Coded Blocks apart from the current
Aggregated stats
> --------------------------------------------------------------------------------------------------------
>                 Key: HDFS-10999
>                 URL: https://issues.apache.org/jira/browse/HDFS-10999
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Wei-Chiu Chuang
>            Assignee: Manoj Govindassamy
>              Labels: hdfs-ec-3.0-nice-to-have, supportability
>         Attachments: HDFS-10999.01.patch
> Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic term "low
redundancy" to the old-fashioned "under replicated". But this term is still being used in
messages in several places, such as web ui, dfsadmin and fsck. We should probably change them
to avoid confusion.
> File this jira to discuss it.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message