Return-Path: 
 <hdfs-user-return-1170-apmail-hadoop-hdfs-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 71E124F40
	for <apmail-hadoop-hdfs-user-archive@minotaur.apache.org>;
 Fri, 13 May 2011 17:41:21 +0000 (UTC)
Received: (qmail 29683 invoked by uid 500); 13 May 2011 17:41:20 -0000
Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org
Received: (qmail 29628 invoked by uid 500); 13 May 2011 17:41:20 -0000
Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-user@hadoop.apache.org>
List-Id: <hdfs-user.hadoop.apache.org>
Reply-To: hdfs-user@hadoop.apache.org
Delivered-To: mailing list hdfs-user@hadoop.apache.org
Received: (qmail 29620 invoked by uid 99); 13 May 2011 17:41:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:41:20 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [65.212.71.225] (HELO prtmm03.dmz.weather.com) (65.212.71.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:41:12 +0000
Received: from ATLMAIL01.corp.weather.com (Not Verified[10.31.24.101]) by
 prtmm03.dmz.weather.com with MailMarshal (v6,8,3,9481)
	id <B4dcd6d220000>; Fri, 13 May 2011 13:40:50 -0400
Received: from PRTCAS01.corp.weather.com ([10.12.25.240]) by
 ATLMAIL01.corp.weather.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 13 May 2011 13:40:50 -0400
Received: from PRTCAS02.corp.weather.com (10.12.25.241) by
 PRTCAS01.corp.weather.com (10.12.25.240) with Microsoft SMTP Server (TLS) id
 14.1.218.12; Fri, 13 May 2011 13:40:50 -0400
Received: from PRTMB01.corp.weather.com ([169.254.2.8]) by
 PRTCAS02.corp.weather.com ([::1]) with mapi id 14.01.0218.012; Fri, 13 May
 2011 13:40:50 -0400
From: "Kester, Scott" <SKester@weather.com>
To: "hdfs-user@hadoop.apache.org" <hdfs-user@hadoop.apache.org>
Subject: Rapid growth in Non DFS Used disk space
Thread-Topic: Rapid growth in Non DFS Used disk space
Thread-Index: AQHMEZTljK7nr+7uZ0evKHhFumQf8g==
Date: Fri, 13 May 2011 17:40:49 +0000
Message-ID: <C9F2E561.76B1%skester@weather.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-MacOutlook/14.10.0.110310
x-originating-ip: [10.12.25.5]
Content-Type: multipart/alternative;
	boundary="_000_C9F2E56176B1skesterweathercom_"
MIME-Version: 1.0
X-OriginalArrivalTime: 13 May 2011 17:40:50.0444 (UTC)
 FILETIME=[E65ABCC0:01CC1194]
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C9F2E56176B1skesterweathercom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

We have an 11 node Hadoop cluster running 20.2 that has been in production =
for 15 months now.  The system is used to process log files that are ingest=
ed daily, and the oldest files in the HDFS are deleted to free up space as =
needed, typically when the free space is less than 10% (the delete is done =
using 'hadoop fs -rmr' on the parent directory of the files to be deleted).=
  When the HDFS was originally built it had 1TB of 'Non DFS' space out of t=
he 20TB total.  This 1TB stayed constant for at least the first year the sy=
stem has been in use.

However over the last few weeks I have seen the 'Non DFS Used' as reported =
by the NameNode dfshealth.jsp page grow to 2G and rising.  The total number=
 of files/directories and blocks in use has remained fairly constant over t=
his time.  I am concerned that the Non DFS Used is going to consume more an=
d more of the HDFS if left unchecked.  Running fcsk gave "The filesystem un=
der path '/' is HEALTHY".

Questions:

A) What exactly is hadoop reporting as 'Non DFS Used', and how is it calcul=
ated?  Are these files on the same partition(s) as the HDFS files, but are =
not actually part of the HDFS?

2) Any ideas on what is driving the growth in Non DFS Used space?   I looke=
d for things like growing log files on the datanodes but didn't find anythi=
ng.

Thanks,
Scott


--_000_C9F2E56176B1skesterweathercom_
Content-Type: text/html; charset="us-ascii"
Content-ID: <0F46E9F8C5E73F4E801FAF9EE5F56711@weather.com>
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
</head>
<body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin=
e-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-fami=
ly: Calibri, sans-serif; ">
<div>We have an 11 node Hadoop cluster running 20.2 that has been in produc=
tion for 15 months now. &nbsp;The system is used to process log files that =
are ingested daily, and the oldest files in the HDFS are deleted to free up=
 space as needed, typically when the
 free space is less than 10% (the delete is done using 'hadoop fs -rmr' on =
the parent directory of the files to be deleted).&nbsp;&nbsp;When the HDFS =
was originally built it had 1TB of 'Non DFS' space out of the 20TB total. &=
nbsp;This 1TB stayed constant for at least the
 first year the system has been in use. &nbsp;&nbsp;</div>
<div><br>
</div>
<div>However over the last few weeks I have seen the 'Non DFS Used' as repo=
rted by the NameNode dfshealth.jsp page grow to 2G and rising. &nbsp;The to=
tal number of files/directories and blocks in use has remained fairly const=
ant over this time. &nbsp;I am concerned that
 the Non DFS Used is going to consume more and more of the HDFS if left unc=
hecked. &nbsp;Running fcsk gave &quot;The filesystem under path '/' is HEAL=
THY&quot;.</div>
<div><br>
</div>
<div>Questions:</div>
<div><br>
</div>
<div>A) What exactly is hadoop reporting as 'Non DFS Used', and how is it c=
alculated? &nbsp;Are these files on the same partition(s) as the HDFS files=
, but are not actually part of the HDFS?&nbsp;</div>
<div><br>
</div>
<div>2) Any ideas on what is driving the growth in Non DFS Used space? &nbs=
p;&nbsp;I looked for things like growing log files on the datanodes but did=
n't find anything.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Scott</div>
<div><br>
</div>
</body>
</html>

--_000_C9F2E56176B1skesterweathercom_--