hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots
Date Thu, 14 Apr 2016 06:15:26 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Xiao Chen updated HDFS-8986:
----------------------------
    Attachment: HDFS-8986.01.patch

I'm attaching a preliminary patch 1 to show the general idea of this change.

I think Chris' comment of modifying {{-count}} together with {{-du}} makes perfect sense,
since they are the 2 usages of {{FileSystem#getContentSummary}} in the shell. (Except for
{{-rm}}'s internal usage, which is not in the scope of this exclude snapshot discussion.)

The general idea is to add a {{-x}} flag to the shell commands, to exclude snapshots from
calculation.

Implementation-wise, since {{getContentSummary}} is a public API on {{FileSystem}}, I don't
think we can change anything there. Alternatively, I added fields and methods to {{ContentSummary}}
(which is {{InterfaceStability.Evolving}} - and I think the changes here doesn't break compatibility
anyway), to store snapshot-related values in the same object. Then at the caller of {{getContentSummary}},
one can subtract the snapshot-related values from the total, to get the desired {{-x}} result.
The calculation is done by adding a snapshot specific {{ContentCounts}} to the {{ContentSummaryComputationContext}}
object.

Please review and provide feedback regarding this approach. I plan to polish the tests/docs
in a later rev. Thanks very much!

> Add option to -du to calculate directory space usage excluding snapshots
> ------------------------------------------------------------------------
>
>                 Key: HDFS-8986
>                 URL: https://issues.apache.org/jira/browse/HDFS-8986
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: snapshots
>            Reporter: Gautam Gopalakrishnan
>            Assignee: Xiao Chen
>         Attachments: HDFS-8986.01.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its children), the
report includes space consumed by blocks that are only present in the snapshots. This is confusing
for end users.
> {noformat}
> $  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0  0  /tmp/parent
> 0  0  /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related disk usage
in the output



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message