hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9230) Report space overhead of unfinalized upgrade/rollingUpgrade
Date Sat, 17 Oct 2015 00:12:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961561#comment-14961561

Andrew Wang commented on HDFS-9230:

For hardlink upgrades, you could check the link count to see if a file in previous is still
referenced in current. This is similar in cost to du.

> Report space overhead of unfinalized upgrade/rollingUpgrade
> -----------------------------------------------------------
>                 Key: HDFS-9230
>                 URL: https://issues.apache.org/jira/browse/HDFS-9230
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: HDFS
>            Reporter: Xiaoyu Yao
> DataNodes do not delete block files during upgrades to allow rollback. This is often
confusing to administrators since they sometimes delete files before finalize upgrade but
don't see the DFS used space reduce.
> Ideally, HDFS should report the un-finalized upgrade overhead along with its message
on NN UI "Upgrade in progress. Not yet finalized." Or, this can be improve with better NN
UI message and document that space won't be reclaimed for deletion until upgrade is finalized.
> For non-rolling upgrade, it is not easy to track this due to hard link. Say NN initialized
upgrade at T1, the block files on DNs that exist before T1 are still under 'current' directory
but is just a hard link to 'previous' directory. When those files are deleted after T1 due
to deletion, the block file usage on DN won't get deleted until upgrade is finalized. 
> So we need to book keeping files created before T1 but deleted after T1 as the un-finalized
upgrade overhead here.
> For rolling upgrade, it is relative easy to track space overhead as we are not using
hard link.

This message was sent by Atlassian JIRA

View raw message