Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 11 Nov 2017 02:22:01 +0000 (UTC)
From: "Virajith Jalaparti (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13116088.1509752611000.211217.1510366921728@Atlassian.JIRA>
In-Reply-To: <JIRA.13116088.1509752611000@Atlassian.JIRA>
References: <JIRA.13116088.1509752611000@Atlassian.JIRA> <JIRA.13116088.1509752611507@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HDFS-12775) [READ] Fix reporting of Provided
 volumes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 11 Nov 2017 02:22:09 -0000


     [ https://issues.apache.org/jira/browse/HDFS-12775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Virajith Jalaparti updated HDFS-12775:
--------------------------------------
    Attachment: HDFS-12775-HDFS-9806.001.patch

Posting a patch that reports the capacity of PROVIDED volumes as follows:
#The capacity (and dfs used) of a PROVIDED volume on a DN is reported to be equal to the total size of the data (in bytes) mounted from the remote storage. Each volume reports zero available capacity (thus 100% usage). This included changes to {{ProvidedVolumeImpl}}, and adding a default {{ProvidedVolumeDF}} implementation and removing the earlier configurable {{ProvidedVolumeDF}} interface.
# The capacity of the Provided volumes is not aggregated in the NN, and does not account towards the total capacity reported by the NN. Thus, the "Configured Capacity" metric reported in the NN Web UI only reports the local capacity available.
# In stats reported by {{BlockStatsMXBean}}, the capacity of the PROVIDED storagetype is not aggregated across Datanodes. The reported capacity is equal to the capacity of the remote storage. 
# Adds a Provided capacity metric to JMX ({{getProvidedCapacity}} and to the NN web UI, to distinguish this capacity from that of the local HDFS capacity.

These changes are motivated by the fact that the capacity of the PROVIDED volumes is virtual, and aggregating it across Datanodes in the Namenode can provide the illusion of having a capacity far greater than what is physically available in the cluster.

> [READ] Fix reporting of Provided volumes
> ----------------------------------------
>
>                 Key: HDFS-12775
>                 URL: https://issues.apache.org/jira/browse/HDFS-12775
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Virajith Jalaparti
>            Assignee: Virajith Jalaparti
>         Attachments: HDFS-12775-HDFS-9806.001.patch
>
>
> Provided Volumes currently report infinite capacity and 0 space used. Further, PROVIDED locations are reported as {{/default-rack/null:0}} in fsck. This JIRA is for making this more readable, and replace these with what users would expect.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org