hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Virajith Jalaparti (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-12775) [READ] Fix reporting of Provided volumes
Date Sat, 11 Nov 2017 02:22:01 GMT

     [ https://issues.apache.org/jira/browse/HDFS-12775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Virajith Jalaparti updated HDFS-12775:
--------------------------------------
    Attachment: HDFS-12775-HDFS-9806.001.patch

Posting a patch that reports the capacity of PROVIDED volumes as follows:
#The capacity (and dfs used) of a PROVIDED volume on a DN is reported to be equal to the total
size of the data (in bytes) mounted from the remote storage. Each volume reports zero available
capacity (thus 100% usage). This included changes to {{ProvidedVolumeImpl}}, and adding a
default {{ProvidedVolumeDF}} implementation and removing the earlier configurable {{ProvidedVolumeDF}}
interface.
# The capacity of the Provided volumes is not aggregated in the NN, and does not account towards
the total capacity reported by the NN. Thus, the "Configured Capacity" metric reported in
the NN Web UI only reports the local capacity available.
# In stats reported by {{BlockStatsMXBean}}, the capacity of the PROVIDED storagetype is not
aggregated across Datanodes. The reported capacity is equal to the capacity of the remote
storage. 
# Adds a Provided capacity metric to JMX ({{getProvidedCapacity}} and to the NN web UI, to
distinguish this capacity from that of the local HDFS capacity.

These changes are motivated by the fact that the capacity of the PROVIDED volumes is virtual,
and aggregating it across Datanodes in the Namenode can provide the illusion of having a capacity
far greater than what is physically available in the cluster.

> [READ] Fix reporting of Provided volumes
> ----------------------------------------
>
>                 Key: HDFS-12775
>                 URL: https://issues.apache.org/jira/browse/HDFS-12775
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Virajith Jalaparti
>            Assignee: Virajith Jalaparti
>         Attachments: HDFS-12775-HDFS-9806.001.patch
>
>
> Provided Volumes currently report infinite capacity and 0 space used. Further, PROVIDED
locations are reported as {{/default-rack/null:0}} in fsck. This JIRA is for making this more
readable, and replace these with what users would expect.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message