Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 56D14EB72 for ; Wed, 23 Jan 2013 16:59:28 +0000 (UTC) Received: (qmail 23523 invoked by uid 500); 23 Jan 2013 16:59:22 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 23403 invoked by uid 500); 23 Jan 2013 16:59:22 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 23371 invoked by uid 99); 23 Jan 2013 16:59:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2013 16:59:22 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates 209.85.215.46 as permitted sender) Received: from [209.85.215.46] (HELO mail-la0-f46.google.com) (209.85.215.46) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2013 16:59:17 +0000 Received: by mail-la0-f46.google.com with SMTP id fq12so6592420lab.19 for ; Wed, 23 Jan 2013 08:58:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:x-gm-message-state; bh=yAaDlMeHdmZ9Y2hnkesmqP1TXsQj6zEWebGAo0Q6IaE=; b=iQUtGOluG5GE43OywyTmpb+cPDQ/arHm+AKP2wkMr33I8dzPAj4m+KA9kUNUyjY00g NE+ZsZnG/zHo3Rb6aPjOmRcyklW7/+wMDb3mHjKGY+RF3e+rQPomJm3Gu9+S8S6E1NPV UX30UVPlw/p6Ck3+3mAOsBuWdPlEOkVzuSI38CuXeoQshaXNX1aRd96N0lLB/jteuVbG TDrLR6Pou1DcjBcC3RM9Ix6EtB7RnfhwKx7XLwEC/hweBKvPK7FkS/odM7jCtDm5oYQQ U+thxBOsqzZkwNJ4oob+0NlO9dLs1XcZQ0aCL3Xi5uxKXf7KOnkpDp2fVoC9dBnrtJ1j tlMg== X-Received: by 10.112.41.202 with SMTP id h10mr1009189lbl.20.1358960335623; Wed, 23 Jan 2013 08:58:55 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.155.229 with HTTP; Wed, 23 Jan 2013 08:58:35 -0800 (PST) In-Reply-To: References: From: Harsh J Date: Wed, 23 Jan 2013 22:28:35 +0530 Message-ID: Subject: Re: NameNode low on available disk space To: Mohit Vadhera Cc: "" Content-Type: multipart/alternative; boundary=e0cb4efa6dfc1196c304d3f79c0f X-Gm-Message-State: ALoCoQmV3Z2j2NuAxdxZkQJcCAl67RQ1IfYkxpkfT7Uq09YK+gRc1s6+qnebJIgpb6sW3l33YFCK X-Virus-Checked: Checked by ClamAV on apache.org --e0cb4efa6dfc1196c304d3f79c0f Content-Type: text/plain; charset=ISO-8859-1 The logs display it in simple bytes. If the issue begins to occur when you start using Hadoop, then its most certainly MR using up the disk space temporarily. You could lower the threshold, or you could perhaps use a bigger disk for your trials/more nodes. On Wed, Jan 23, 2013 at 10:25 PM, Mohit Vadhera < project.linux.proj@gmail.com> wrote: > MR operation are running on the same machine. i checked the parameter " > mapred.local.dir" in my installed directory /etc/hadoop/ but didn't find . > One question the disk space reserved size displayed in logs in KB or MB ? > I am layman on hadoop. The link I followed to install is given below > > > https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode > > Thanks, > > > > > On Wed, Jan 23, 2013 at 10:12 PM, Harsh J wrote: > >> A random switching behavior can only be explained by a fluctuating disk >> space I'd think. Are you running MR operations on the same disk (i.e. is it >> part of mapred.local.dir as well)? >> >> >> On Wed, Jan 23, 2013 at 9:24 PM, Mohit Vadhera < >> project.linux.proj@gmail.com> wrote: >> >>> NN switches randomly into the safemode then I run command to leave >>> safemode manually. I never got alerts for low disk space on machine level >>> and i didn't see the space fluctuates GBs into MBs . >>> >>> >>> >>> >>> >>> On Wed, Jan 23, 2013 at 9:10 PM, Harsh J wrote: >>> >>>> Mohit, >>>> >>>> When do you specifically get the error at the NN? Does your NN >>>> consistently not start with that error? >>>> >>>> Your local disk space availability can certainly fluctuate if you use >>>> the same disk for MR and other activity which creates temporary files. >>>> >>>> >>>> On Wed, Jan 23, 2013 at 9:01 PM, Mohit Vadhera < >>>> project.linux.proj@gmail.com> wrote: >>>> >>>>> Can somebody answer me on this plz ? >>>>> >>>>> >>>>> On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera < >>>>> project.linux.proj@gmail.com> wrote: >>>>> >>>>>> Thanks Guys, As you said the level is already pretty low i.e 100 MB >>>>>> but in my case the root fs / has 14 G available. What can be the root >>>>>> cause then ? >>>>>> >>>>>> /dev/mapper/vg_operamast1-lv_root >>>>>> 50G 33G 14G 71% / >>>>>> >>>>>> As per logs. >>>>>> 2013-01-21 01:22:52,217 WARN >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>>>>> available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which >>>>>> is below the configured reserved amount 104857600 >>>>>> >>>>>> >>>>>> On Wed, Jan 23, 2013 at 11:13 AM, Harsh J wrote: >>>>>> >>>>>>> Hi again, >>>>>>> >>>>>>> Yes, you need to add it to hdfs-site.xml and restart the NN. >>>>>>> >>>>>>> > Thanks Harsh, Do I need to add parameters in hdfs-site.xml and >>>>>>> restart service namenode. >>>>>>> > + public static final String DFS_NAMENODE_DU_RESERVED_KEY = >>>>>>> "dfs.namenode.resource.du. >>>>>>> reserved"; >>>>>>> > + public static final long DFS_NAMENODE_DU_RESERVED_DEFAULT = >>>>>>> 1024 * 1024 * 100; // 100 MB >>>>>>> >>>>>>> >>>>>>> On Wed, Jan 23, 2013 at 10:12 AM, Harsh J wrote: >>>>>>> >>>>>>>> Edit your hdfs-site.xml (or whatever place of config your NN uses) >>>>>>>> to lower the value of property "dfs.namenode.resource.du.reserved". Create >>>>>>>> a new property if one does not exist, and set the value of space to a >>>>>>>> suitable level. The default itself is pretty low - 100 MB in bytes. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera < >>>>>>>> project.linux.proj@gmail.com> wrote: >>>>>>>> >>>>>>>>> Ok Steve. I am forwarding my issue again to the list that you >>>>>>>>> said. The version is >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Namenode switches into safemode when it has low disk space on the >>>>>>>>> root fs / i have to manually run a command to leave it. Below are log >>>>>>>>> messages for low space on root / fs. Is there any parameter so that i can >>>>>>>>> reduce reserved amount.Hadoop 2.0.0-cdh4.1.2 >>>>>>>>> >>>>>>>>> 2013-01-21 01:22:52,217 WARN >>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>>>>>>>> available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below >>>>>>>>> the configured reserved amount 104857600 >>>>>>>>> 2013-01-21 01:22:52,218 WARN >>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on >>>>>>>>> available disk space. Entering safe mode. >>>>>>>>> 2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: >>>>>>>>> STATE* Safe mode is ON. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran < >>>>>>>>> steve.loughran@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> ser@hadoop.apache.orglist >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Harsh J >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Harsh J >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Harsh J >>>> >>> >>> >> >> >> -- >> Harsh J >> > > -- Harsh J --e0cb4efa6dfc1196c304d3f79c0f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
The logs display it in simple bytes. If the issue begins t= o occur when you start using Hadoop, then its most certainly MR using up th= e disk space temporarily.

You could lower the thre= shold, or you could perhaps use a bigger disk for your trials/more nodes.


On Wed,= Jan 23, 2013 at 10:25 PM, Mohit Vadhera <project.linux.proj@gm= ail.com> wrote:
MR operation are runni= ng on the same machine. i checked the parameter " mapred.local.dir&quo= t; in my installed directory /etc/hadoop/ but didn't find . One questio= n the disk space reserved size displayed in logs in KB or MB ?=A0 I am laym= an on hadoop. The link I followed to install is given below

https://c= cp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+P= seudo-distributed+Mode

Thanks,




On= Wed, Jan 23, 2013 at 10:12 PM, Harsh J <harsh@cloudera.com> wrote:
A random switching behavior= can only be explained by a fluctuating disk space I'd think. Are you r= unning MR operations on the same disk (i.e. is it part of mapred.local.dir = as well)?


On Wed, Jan 23, 2013 at 9:24 PM, Mohit V= adhera <project.linux.proj@gmail.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
NN switches randomly into the safemode then I run com= mand to leave safemode manually. I never got alerts for low disk space on m= achine level and i didn't see the space fluctuates GBs into MBs .





On Wed, Jan 23, 2013 at 9:10 PM, Harsh J <harsh@clou= dera.com> wrote:
Mohit,

W= hen do you specifically get the error at the NN? Does your NN consistently = not start with that error?

Your local disk space availability can certainly fluctu= ate if you use the same disk for MR and other activity which creates tempor= ary files.


On Wed, Jan 23, 2013 at 9:01 PM, Mohit Vadhera <project.linu= x.proj@gmail.com> wrote:
Can somebody answer me on t= his plz ?


On Wed, Jan 23, 2013 at 11:44 AM, Mohit = Vadhera <project.linux.proj@gmail.com> wrote:
Thanks Guys, As you said th= e level is already pretty low i.e 100 MB but=20 in my case the root fs / has 14 G available.=A0 What can be the root cause then ?

/dev/mapper/vg_operamast1-lv_root
=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 50G=A0=A0 33G=A0=A0 14G=A0 71= % /

As per logs.
=A0=A0=A0 2013-01-21 01:22:52,217 WARN=20 org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space=20 available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696= ,=20 which is below the configured reserved amount 104857600
=


On Wed, Jan 2= 3, 2013 at 11:13 AM, Harsh J <harsh@cloudera.com> wrote:
Hi again,

Yes, you need to add it to hdfs-site.xml and restart the NN.

> Tha= nks Harsh, Do I need to add parameters in hdfs-site.xml and restart service= namenode.
> + =A0publi= c static final String =A0DFS_NAMENODE_DU_RESERVED_KEY =3D "dfs.namenod= e.resource.du.
reserved";
> + =A0public static final long =A0 =A0DFS_NAMENODE_DU_RESERVED_DEFAULT = =3D 1024 * 1024 * 100; // 100 MB

=
On Wed, Jan 23, 2013 at 10:12 AM, Harsh J <harsh@cloudera.com> wrote:
Edit your hdfs-site.xml (or= whatever place of config your NN uses) to lower the value of property &quo= t;dfs.namenode.resource.du.reserved". Create a new property if one doe= s not exist, and set the value of space to a suitable level. The default it= self is pretty low - 100 MB in bytes.


On Wed, Jan 23, 2013 at 9:13 AM, Mohit V= adhera <project.linux.proj@gmail.com> wrote:
= Ok Steve. I am forwarding my issue again to the list that you said. The ver= sion is

Hi,

Namenode switches into safemode when it has low disk space on the=20 root fs / i have to manually run a command to leave it. Below are log=20 messages for low space on root / fs. Is there any parameter so that i=20 can reduce reserved amount.Hadoop 2.0.0-cdh4.1.2

2013-01-21 01:22:52,217 WARN=20 org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space=20 available on volume '/dev/mapper/vg_lv_root' is 10653696, which is = below the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN=20 org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on=20 available disk space. Entering safe mode.
2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: STATE* Saf= e mode is ON.




On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <steve.loughran@gmail.com> wrote:
ser@hadoop.apache.orglist=





<= /div>--
Har= sh J



--
Harsh J





<= /div>--
Harsh J




<= /div>--
Harsh J




--
= Harsh J --e0cb4efa6dfc1196c304d3f79c0f--