Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: 
 <DM2PR0501MB1647D4B73102830BA30FC71DC0E50@DM2PR0501MB1647.namprd05.prod.outlook.com>
References: 
 <DM2PR0501MB1647D4B73102830BA30FC71DC0E50@DM2PR0501MB1647.namprd05.prod.outlook.com>
From: Namikaze Minato <lloydsensei@gmail.com>
Date: Tue, 22 Dec 2015 03:24:41 +0100
Message-ID: 
 <CACmJb3ySLNuy72Y+rL=oz==sx0JOv37FGx0tdNmCQePU6n08rw@mail.gmail.com>
Subject: Re: diagnosing the difference between dfs 'du' and 'df'
To: Martin Serrano <martin@attivio.com>
Cc: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: text/plain; charset=UTF-8

This may be a wrong lead, but try to do your "du" command as hdfs
user, so that we are sure that we don't miss out read-protected
folders.

Regards,
LLoyd

On 22 December 2015 at 03:21, Martin Serrano <martin@attivio.com> wrote:
> Hi,
>
> I have an application that is writing data rapidly directly to HDFS
> (creates and appends) as well as to HBase (10-15 tables).  The disk free
> for the filesystem will report that a large percentage of the system is
> in use:
>
> $ hdfs dfs -df -h /
> Filesystem     Size     Used  Available  Use%
> hdfs://ha   882.6 G  472.6 G    409.9 G   54%
>
> Yet when I try to figure out where the disk space is being used using
> dfs -du reports:
>
> $ hdfs dfs -du -h /
> 0        /app-logs
> 7.6 G    /apps
> 382.2 M  /hdp
> 0        /mapred
> 0        /mr-history
> 8.5 K    /tmp
> 3.8 G    /user
>
> A dfsadmin -report during the same time frame is below.  I'm trying to
> figure out where all of this space is going to.  When my application is
> killed or quiescent, the df and dfsadmin reports fall in line with what
> I would expect.  I'm running HDP 2.3 with a default configuration as set
> up by Ambari.  I'm looking for hints or suggestions on how I can
> investigate this issue.  It seems crazy that ingesting 12g or so of data
> can temporarily consume (reserve?) ~300g of HDFS.
>
> Thanks,
> Martin
>
> Configured Capacity: 947644268544 (882.56 GB)
> Present Capacity: 947064596261 (882.02 GB)
> DFS Remaining: 490046627240 (456.39 GB)
> DFS Used: 457017969021 (425.63 GB)
> DFS Used%: 48.26%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
>
> -------------------------------------------------
> Live datanodes (3):
>
> Name: *.*.*.*:50010 (**********.com)
> Hostname: **********.com
> Decommission Status : Normal
> Configured Capacity: 315881422848 (294.19 GB)
> DFS Used: 218955099179 (203.92 GB)
> Non DFS Used: 168255175 (160.46 MB)
> DFS Remaining: 96758068494 (90.11 GB)
> DFS Used%: 69.32%
> DFS Remaining%: 30.63%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 15
> Last contact: Mon Dec 21 17:17:38 EST 2015
>
>
> Name: *.*.*.*:50010 (**********.com)
> Hostname: **********.com
> Decommission Status : Normal
> Configured Capacity: 315881422848 (294.19 GB)
> DFS Used: 218873337575 (203.84 GB)
> Non DFS Used: 151608508 (144.59 MB)
> DFS Remaining: 96856476765 (90.20 GB)
> DFS Used%: 69.29%
> DFS Remaining%: 30.66%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 16
> Last contact: Mon Dec 21 17:17:38 EST 2015
>
>
> Name: *.*.*.*:50010 (*************.com)
> Hostname: ***********.com
> Decommission Status : Normal
> Configured Capacity: 315881422848 (294.19 GB)
> DFS Used: 19189532267 (17.87 GB)
> Non DFS Used: 259808600 (247.77 MB)
> DFS Remaining: 296432081981 (276.07 GB)
> DFS Used%: 6.07%
> DFS Remaining%: 93.84%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 16
> Last contact: Mon Dec 21 17:17:39 EST 2015
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org