Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
From: Martin Serrano <martin@attivio.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: diagnosing the difference between dfs 'du' and 'df'
Thread-Topic: diagnosing the difference between dfs 'du' and 'df'
Thread-Index: AQHRPF9161gbVVsZeEynlYF1dRh3DA==
Date: Tue, 22 Dec 2015 02:21:25 +0000
Message-ID: 
 <DM2PR0501MB1647D4B73102830BA30FC71DC0E50@DM2PR0501MB1647.namprd05.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: attivio.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:23
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Dec 2015 02:21:25.6215
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: f2451f17-698f-4d48-b34a-740e2ecaa55e
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR0501MB1645

Hi,=0A=
=0A=
I have an application that is writing data rapidly directly to HDFS=0A=
(creates and appends) as well as to HBase (10-15 tables).  The disk free=0A=
for the filesystem will report that a large percentage of the system is=0A=
in use:=0A=
=0A=
$ hdfs dfs -df -h /=0A=
Filesystem     Size     Used  Available  Use%=0A=
hdfs://ha   882.6 G  472.6 G    409.9 G   54%=0A=
=0A=
Yet when I try to figure out where the disk space is being used using=0A=
dfs -du reports:=0A=
=0A=
$ hdfs dfs -du -h /=0A=
0        /app-logs=0A=
7.6 G    /apps=0A=
382.2 M  /hdp=0A=
0        /mapred=0A=
0        /mr-history=0A=
8.5 K    /tmp=0A=
3.8 G    /user=0A=
=0A=
A dfsadmin -report during the same time frame is below.  I'm trying to=0A=
figure out where all of this space is going to.  When my application is=0A=
killed or quiescent, the df and dfsadmin reports fall in line with what=0A=
I would expect.  I'm running HDP 2.3 with a default configuration as set=0A=
up by Ambari.  I'm looking for hints or suggestions on how I can=0A=
investigate this issue.  It seems crazy that ingesting 12g or so of data=0A=
can temporarily consume (reserve?) ~300g of HDFS.=0A=
=0A=
Thanks,=0A=
Martin=0A=
=0A=
Configured Capacity: 947644268544 (882.56 GB)=0A=
Present Capacity: 947064596261 (882.02 GB)=0A=
DFS Remaining: 490046627240 (456.39 GB)=0A=
DFS Used: 457017969021 (425.63 GB)=0A=
DFS Used%: 48.26%=0A=
Under replicated blocks: 0=0A=
Blocks with corrupt replicas: 0=0A=
Missing blocks: 0=0A=
Missing blocks (with replication factor 1): 0=0A=
=0A=
-------------------------------------------------=0A=
Live datanodes (3):=0A=
=0A=
Name: *.*.*.*:50010 (**********.com)=0A=
Hostname: **********.com=0A=
Decommission Status : Normal=0A=
Configured Capacity: 315881422848 (294.19 GB)=0A=
DFS Used: 218955099179 (203.92 GB)=0A=
Non DFS Used: 168255175 (160.46 MB)=0A=
DFS Remaining: 96758068494 (90.11 GB)=0A=
DFS Used%: 69.32%=0A=
DFS Remaining%: 30.63%=0A=
Configured Cache Capacity: 0 (0 B)=0A=
Cache Used: 0 (0 B)=0A=
Cache Remaining: 0 (0 B)=0A=
Cache Used%: 100.00%=0A=
Cache Remaining%: 0.00%=0A=
Xceivers: 15=0A=
Last contact: Mon Dec 21 17:17:38 EST 2015=0A=
=0A=
=0A=
Name: *.*.*.*:50010 (**********.com)=0A=
Hostname: **********.com=0A=
Decommission Status : Normal=0A=
Configured Capacity: 315881422848 (294.19 GB)=0A=
DFS Used: 218873337575 (203.84 GB)=0A=
Non DFS Used: 151608508 (144.59 MB)=0A=
DFS Remaining: 96856476765 (90.20 GB)=0A=
DFS Used%: 69.29%=0A=
DFS Remaining%: 30.66%=0A=
Configured Cache Capacity: 0 (0 B)=0A=
Cache Used: 0 (0 B)=0A=
Cache Remaining: 0 (0 B)=0A=
Cache Used%: 100.00%=0A=
Cache Remaining%: 0.00%=0A=
Xceivers: 16=0A=
Last contact: Mon Dec 21 17:17:38 EST 2015=0A=
=0A=
=0A=
Name: *.*.*.*:50010 (*************.com)=0A=
Hostname: ***********.com=0A=
Decommission Status : Normal=0A=
Configured Capacity: 315881422848 (294.19 GB)=0A=
DFS Used: 19189532267 (17.87 GB)=0A=
Non DFS Used: 259808600 (247.77 MB)=0A=
DFS Remaining: 296432081981 (276.07 GB)=0A=
DFS Used%: 6.07%=0A=
DFS Remaining%: 93.84%=0A=
Configured Cache Capacity: 0 (0 B)=0A=
Cache Used: 0 (0 B)=0A=
Cache Remaining: 0 (0 B)=0A=
Cache Used%: 100.00%=0A=
Cache Remaining%: 0.00%=0A=
Xceivers: 16=0A=
Last contact: Mon Dec 21 17:17:39 EST 2015=0A=
=0A=
=0A=

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org