Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of sagar_shukla@persistent.co.in
 designates 202.54.11.87 as permitted sender)
From: Sagar Shukla <sagar_shukla@persistent.co.in>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Fri, 8 Jul 2011 16:54:24 +0530
Subject: RE: Difference between DFS Used and Non-DFS Used
Thread-Topic: Difference between DFS Used and Non-DFS Used
Thread-Index: Acw9X/xQnMsvK9LsSbq7uCzhC4jz8QAAUohA
Message-ID: 
 <D1B70AEEBAE689468C0E7DE40BEE854A454AFCEBEC@EXCHANGE.persistent.co.in>
References: 
 <D1B70AEEBAE689468C0E7DE40BEE854A454AEF4E9A@EXCHANGE.persistent.co.in>
 <CAOcnVr1zzByQvjvtM6CcvWRxsX6tOr413BMjCqUTYF9WxNUopg@mail.gmail.com>
 <D1B70AEEBAE689468C0E7DE40BEE854A454AF56E16@EXCHANGE.persistent.co.in>
 <CAOcnVr1xYgar=_bM4yzm42D7VGNYH6DrhcFrsfA=i8_zXKaZ9A@mail.gmail.com>
In-Reply-To: 
 <CAOcnVr1xYgar=_bM4yzm42D7VGNYH6DrhcFrsfA=i8_zXKaZ9A@mail.gmail.com>
Accept-Language: en-US, en-IN
Content-Language: en-US
acceptlanguage: en-US, en-IN
Content-Type: text/plain; charset="iso-8859-1"
content-transfer-encoding: quoted-printable
MIME-Version: 1.0

Thanks Harsh. My first question still remains unanswered - "Why does it requ=
ire non-DFS storage?". If it is cache data then it should get flushed from t=
he system after certain interval of time. And if it is useful data then it s=
hould have been part of used DFS data.

I have a setup in which DFS used is use approx. 10 MB whereas non-DFS used i=
s around 250 GB which is quite ridiculous.

Thanks,
Sagar

-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com] 
Sent: Friday, July 08, 2011 4:42 PM
To: common-user@hadoop.apache.org
Subject: Re: Difference between DFS Used and Non-DFS Used

It is just for information's sake (cause it can be computed with the
data collected). The space is accounted just to let you know that
there's something being stored on the DataNodes apart from just the
HDFS data, in case you are running out of space.

On Fri, Jul 8, 2011 at 10:18 AM, Sagar Shukla
<sagar_shukla@persistent.co.in> wrote:
> Hi Harsh,
> =A0 =A0 Thanks for your reply.
>
> But why does it require non-DFS storage ? And why that space is accounted=
 differently from regular DFS storage ?
>
> Ideally, it should have been part of same storage.
>
> Thanks,
> Sagar
>
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Thursday, July 07, 2011 6:04 PM
> To: common-user@hadoop.apache.org
> Subject: Re: Difference between DFS Used and Non-DFS Used
>
> DFS used is a count of all the space used by the dfs.data.dirs. The
> non-dfs used space is whatever space is occupied beyond that (which
> the DN does not account for).
>
> On Thu, Jul 7, 2011 at 3:29 PM, Sagar Shukla
> <sagar_shukla@persistent.co.in> wrote:
>> Hi,
>> =A0 =A0 =A0 What is the difference between DFS Used and Non-DFS used ?
>>
>> Thanks,
>> Sagar
>>
>> DISCLAIMER
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> This e-mail may contain privileged and confidential information which is=
 the property of Persistent Systems Ltd. It is intended only for the use of=
 the individual or entity to which it is addressed. If you are not the inten=
ded recipient, you are not authorized to read, retain, copy, print, distribu=
te or use this message. If you have received this communication in error, pl=
ease notify the sender and delete all copies of this message. Persistent Sys=
tems Ltd. does not accept any liability for virus infected mails.
>>
>>
>
>
>
> --
> Harsh J
>
> DISCLAIMER
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> This e-mail may contain privileged and confidential information which is t=
he property of Persistent Systems Ltd. It is intended only for the use of th=
e individual or entity to which it is addressed. If you are not the intended=
 recipient, you are not authorized to read, retain, copy, print, distribute=
 or use this message. If you have received this communication in error, plea=
se notify the sender and delete all copies of this message. Persistent Syste=
ms Ltd. does not accept any liability for virus infected mails.
>
>


-- 
Harsh J

DISCLAIMER=0A=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A=
This e-mail may contain privileged and confidential information which is the=
 property of Persistent Systems Ltd. It is intended only for the use of the=
 individual or entity to which it is addressed. If you are not the intended=
 recipient, you are not authorized to read, retain, copy, print, distribute=
 or use this message. If you have received this communication in error, plea=
se notify the sender and delete all copies of this message. Persistent Syste=
ms Ltd. does not accept any liability for virus infected mails.=0A=