hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: NameNode low on available disk space
Date Wed, 23 Jan 2013 16:42:54 GMT
A random switching behavior can only be explained by a fluctuating disk
space I'd think. Are you running MR operations on the same disk (i.e. is it
part of mapred.local.dir as well)?


On Wed, Jan 23, 2013 at 9:24 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:

> NN switches randomly into the safemode then I run command to leave
> safemode manually. I never got alerts for low disk space on machine level
> and i didn't see the space fluctuates GBs into MBs .
>
>
>
>
>
> On Wed, Jan 23, 2013 at 9:10 PM, Harsh J <harsh@cloudera.com> wrote:
>
>> Mohit,
>>
>> When do you specifically get the error at the NN? Does your NN
>> consistently not start with that error?
>>
>> Your local disk space availability can certainly fluctuate if you use the
>> same disk for MR and other activity which creates temporary files.
>>
>>
>> On Wed, Jan 23, 2013 at 9:01 PM, Mohit Vadhera <
>> project.linux.proj@gmail.com> wrote:
>>
>>> Can somebody answer me on this plz ?
>>>
>>>
>>> On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera <
>>> project.linux.proj@gmail.com> wrote:
>>>
>>>> Thanks Guys, As you said the level is already pretty low i.e 100 MB but
>>>> in my case the root fs / has 14 G available.  What can be the root cause
>>>> then ?
>>>>
>>>> /dev/mapper/vg_operamast1-lv_root
>>>>                        50G   33G   14G  71% /
>>>>
>>>> As per logs.
>>>>     2013-01-21 01:22:52,217 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
>>>> is below the configured reserved amount 104857600
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 11:13 AM, Harsh J <harsh@cloudera.com> wrote:
>>>>
>>>>> Hi again,
>>>>>
>>>>> Yes, you need to add it to hdfs-site.xml and restart the NN.
>>>>>
>>>>> > Thanks Harsh, Do I need to add parameters in hdfs-site.xml and
>>>>> restart service namenode.
>>>>> > +  public static final String  DFS_NAMENODE_DU_RESERVED_KEY =
>>>>> "dfs.namenode.resource.du.
>>>>> reserved";
>>>>> > +  public static final long    DFS_NAMENODE_DU_RESERVED_DEFAULT
=
>>>>> 1024 * 1024 * 100; // 100 MB
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 10:12 AM, Harsh J <harsh@cloudera.com>
wrote:
>>>>>
>>>>>> Edit your hdfs-site.xml (or whatever place of config your NN uses)
to
>>>>>> lower the value of property "dfs.namenode.resource.du.reserved".
Create a
>>>>>> new property if one does not exist, and set the value of space to
a
>>>>>> suitable level. The default itself is pretty low - 100 MB in bytes.
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
>>>>>> project.linux.proj@gmail.com> wrote:
>>>>>>
>>>>>>> Ok Steve. I am forwarding my issue again to the list that you
said.
>>>>>>> The version is
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Namenode switches into safemode when it has low disk space on
the
>>>>>>> root fs / i have to manually run a command to leave it. Below
are log
>>>>>>> messages for low space on root / fs. Is there any parameter so
that i can
>>>>>>> reduce reserved amount.Hadoop 2.0.0-cdh4.1.2
>>>>>>>
>>>>>>> 2013-01-21 01:22:52,217 WARN
>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
Space
>>>>>>> available on volume '/dev/mapper/vg_lv_root' is 10653696, which
is below
>>>>>>> the configured reserved amount 104857600
>>>>>>> 2013-01-21 01:22:52,218 WARN
>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode
low on
>>>>>>> available disk space. Entering safe mode.
>>>>>>> 2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange:
>>>>>>> STATE* Safe mode is ON.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
>>>>>>> steve.loughran@gmail.com> wrote:
>>>>>>>
>>>>>>>> ser@hadoop.apache.orglist
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Harsh J
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Harsh J
>>
>
>


-- 
Harsh J

Mime
View raw message