Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 27B0717908 for ; Fri, 10 Apr 2015 14:25:35 +0000 (UTC) Received: (qmail 70602 invoked by uid 500); 10 Apr 2015 14:25:26 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 70484 invoked by uid 500); 10 Apr 2015 14:25:26 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 70474 invoked by uid 99); 10 Apr 2015 14:25:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Apr 2015 14:25:26 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nataraj.jonnalagadda@gmail.com designates 209.85.220.51 as permitted sender) Received: from [209.85.220.51] (HELO mail-pa0-f51.google.com) (209.85.220.51) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Apr 2015 14:25:00 +0000 Received: by paboj16 with SMTP id oj16so23292854pab.0 for ; Fri, 10 Apr 2015 07:24:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=HUKB0LTBphNVD1U55nYGjm0VDleE+JkV8xJrHouGK8U=; b=qlGMQf61MiWDNXp/IiiKpWqyyNjCckugI0rBdQbQE3ColrD9uSMPsEmOC61zRn0K2I CS15trODR2RyHCYYF5QeiN8JngzzrWTmxD06OVsw27luMyMIz1ktBBhaSzFWIFyP6apd rWDhoMFr44r7mT0pkMWV2tgUem/K/WZmT7QwDFjQidWgagZSb0xhvDgI6srxKpmp2Xhd qV83s0M/Z+6t0lDapwqQn2D+DFiuIv+qfBSUESBg4zzYvmm8/WKNLi7qmw1aeHgAbn5q ZYdaymGptpVf4lklLuZG37k8kKUQQS5TGmSGx3hsJxbu2frJJE8pcPmN/42flNBgnAWY ACrg== MIME-Version: 1.0 X-Received: by 10.70.29.165 with SMTP id l5mr3075199pdh.125.1428675853387; Fri, 10 Apr 2015 07:24:13 -0700 (PDT) Received: by 10.70.109.167 with HTTP; Fri, 10 Apr 2015 07:24:13 -0700 (PDT) In-Reply-To: References: Date: Fri, 10 Apr 2015 07:24:13 -0700 Message-ID: Subject: Re: hdfs disk usage From: nataraj jonnalagadda To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7bdca3f2bd913905135f84cf X-Virus-Checked: Checked by ClamAV on apache.org --047d7bdca3f2bd913905135f84cf Content-Type: text/plain; charset=ISO-8859-1 Nitin, You need to mount each disk as a location eg: /data/disk01, /data/disk02, /data/disk03 and specify each of these locations (comma separated) in the parameter dfs.datanode.data.dir=/data/data01/hadoop/hdfs/data, /data/data02/hadoop/hdfs/data, /data/data03/hadoop/hdfs/data of the hdfs-site.xml file. Thanks, Nataraj. On Fri, Apr 10, 2015 at 7:10 AM, Nitin Pawar wrote: > I just restarted the cluster and it seems it resolved the problem > > I will repost if this issue comes again > > On Fri, Apr 10, 2015 at 7:35 PM, Nitin Pawar > wrote: > >> Thanks Peyman >> >> I think it is not related to replication. >> >> hdfs dfsadmin is reporting following stats >> Disk Usage (DFS Used)108.4 GB / 782.9 GB (13.85%)Disk Usage (Non DFS >> Used)583.9 GB / 782.9 GB (74.58%) >> >> In the Non DFS used on all the disks, atleast 150GB is available (when i >> do df -h) >> >> This has marked my cluster at 90% and I want to understand why Non DFS >> used is represented so high when its not >> >> On Fri, Apr 10, 2015 at 7:28 PM, Peyman Mohajerian >> wrote: >> >>> Take the default 3x replication into account too. >>> >>> On Fri, Apr 10, 2015 at 6:50 AM, Nitin Pawar >>> wrote: >>> >>>> Hi Guys, >>>> >>>> I have setup a 6 node cluster using hadoop 2.6 out of which 4 are data >>>> nodes. >>>> >>>> Each datanode disk is 200GB (so total storage size of 800 GB) >>>> >>>> But when started, configured dfs storage was only 200GB. >>>> >>>> There are no extra mounted disks or additional directories configured >>>> for each mount. >>>> >>>> Can someone help me on how do i use all available 800GB from 4 data >>>> nodes as hdfs? >>>> >>>> -- >>>> Nitin Pawar >>>> >>> >>> >> >> >> -- >> Nitin Pawar >> > > > > -- > Nitin Pawar > --047d7bdca3f2bd913905135f84cf Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Nitin,

You need to mount each disk as a= location eg: /data/disk01, /data/disk02, /data/disk03 and specify each of = these locations (comma separated) in the parameter dfs.dat= anode.data.dir=3D/data/data01/hadoop/hdfs/da= ta, /data/data02/hadoop/hdfs/data, /data/data03/hadoop/hdfs/data of the hdf= s-site.xml file.

Thanks,
Nataraj.
<= div>


On Fri, Apr= 10, 2015 at 7:10 AM, Nitin Pawar <nitinpawar432@gmail.com> wrote:
I just rest= arted the cluster and it seems it resolved the problem=A0

I will repost if this issue comes again=A0

On Fri, Apr 1= 0, 2015 at 7:35 PM, Nitin Pawar <nitinpawar432@gmail.com> wrote:
Thanks Peyman= =A0

I think it is not related to replication.=A0

hdfs dfsadmin is reporting following stats=A0
<= table style=3D"max-width:100%;border-collapse:collapse;border-spacing:0px;w= idth:470px;margin-bottom:0px;border-top-style:none;color:rgb(102,102,102);f= ont-size:13px;font-family:'Helvetica Neue',Helvetica,Arial,sans-ser= if;line-height:20px;border-right-style:none!important;border-bottom-style:n= one!important;border-left-style:none!important">Disk Usage (DFS Used)108.4 GB / 782.9 GB (13.85%)<= td style=3D"padding:4px 5px;text-align:right;vertical-align:top;border-top-= style:none">Disk Usage (Non DFS Used)583.9 GB / 782.9 GB (74.= 58%)

In the Non DFS used on all= the disks, atleast 150GB is available (when i do df -h)

This has marked my cluster at 90% and I want to understand why Non D= FS used is represented so high when its not=A0

On Fri, Apr 10, 2015 at = 7:28 PM, Peyman Mohajerian <mohajeri@gmail.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
Take the default 3x replicat= ion into account too.

On Fri, Apr 10, 2015 at 6:50 AM, Nitin Pawar <n= itinpawar432@gmail.com> wrote:
Hi Guys,

I have setup a 6 node clus= ter using hadoop 2.6 out of which 4 are data nodes.=A0

=
Each datanode disk is 200GB (so total storage size of 800 GB)

But when started, configured dfs storage was only 200GB.

There are no extra mounted disks or additional dire= ctories configured for each mount.

Can someone hel= p me on how do i use all available 800GB from 4 data nodes as hdfs?=A0

--
Nitin Pawar<= br>




<= /div>--
Nitin Pawar



--
Nitin Pawar
--047d7bdca3f2bd913905135f84cf--