hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Namenode automatically going to safemode with 2.1.0-beta
Date Sat, 20 Jul 2013 00:50:29 GMT
Yeah I believe your /tmp was probably misbehaving somehow (running out
of space or otherwise). You could log a JIRA for the "null" seen in
the log though, it shouldn't have done that and should've shown the
real mount point.

On Fri, Jul 19, 2013 at 8:47 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:
> this is not a bug.
>
> it has been documented.
>
> On Jul 19, 2013 10:13 PM, "Krishna Kishore Bonagiri"
> <write2kishore@gmail.com> wrote:
>>
>> Hi Harsh,
>>
>>   I have made my dfs.namenode.name.dir point to a subdirectory of my home,
>> and I don't see this issue again. So, is this a bug that we need to log into
>> JIRA?
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <harsh@cloudera.com> wrote:
>>>
>>> > 2013-07-12 11:04:26,002 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> > available on volume 'null' is 0, which is below the configured reserved
>>> > amount 104857600
>>>
>>> This is interesting. Its calling your volume "null", which may be more
>>> of a superficial bug.
>>>
>>> What is your dfs.namenode.name.dir set to? From
>>> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
>>> is being used off of the out-of-box defaults. Could you try to set it
>>> to a specific directory thats not on /tmp?
>>>
>>> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
>>> <write2kishore@gmail.com> wrote:
>>> > I don't have it in my hdfs-site.xml, in which case probably the default
>>> > value is taken..
>>> >
>>> >
>>> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>> >>
>>> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
>>> >>
>>> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <adityaexalter@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi Krishna,
>>> >>>
>>> >>>    Can you please send screenshots of namenode web UI.
>>> >>>
>>> >>> Thanks Aditya.
>>> >>>
>>> >>>
>>> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>>> >>> <write2kishore@gmail.com> wrote:
>>> >>>>
>>> >>>> I have had enough space on the disk that is used, like around
30
>>> >>>> Gigs
>>> >>>>
>>> >>>> Thanks,
>>> >>>> Kishore
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>>> >>>> <venkatarami.netla@cloudwick.com> wrote:
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>> pls see the available space for NN storage directory.
>>> >>>>>
>>> >>>>> Thanks & Regards
>>> >>>>>
>>> >>>>> Venkat
>>> >>>>>
>>> >>>>>
>>> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>>> >>>>> <write2kishore@gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> Hi,
>>> >>>>>>
>>> >>>>>>  I am doing no activity on my single node cluster which
is using
>>> >>>>>> 2.1.0-beta, and still observed that it has gone to safe
mode by
>>> >>>>>> itself after
>>> >>>>>> a while. I was looking at the name node log and see
many of these
>>> >>>>>> kinds of
>>> >>>>>> entries.. Can anything be interpreted from these?
>>> >>>>>>
>>> >>>>>> 2013-07-12 09:06:11,256 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting
log
>>> >>>>>> segment at
>>> >>>>>> 561
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling
edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending
log
>>> >>>>>> segment 561
>>> >>>>>> 2013-07-12 09:07:11,291 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number
of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 1 Number of transactions
batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> >>>>>> 2013-07-12 09:07:11,292 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number
of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 1 Number of transactions
batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> >>>>>> 2013-07-12 09:07:11,293 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> >>>>>> 2013-07-12 09:07:11,294 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting
log
>>> >>>>>> segment at
>>> >>>>>> 563
>>> >>>>>> 2013-07-12 09:08:11,397 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:08:11,398 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling
edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:08:11,398 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending
log
>>> >>>>>> segment 563
>>> >>>>>> 2013-07-12 09:08:11,399 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number
of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 2 Number of transactions
batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> >>>>>> 2013-07-12 09:08:11,400 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number
of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 2 Number of transactions
batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 09:08:11,402 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> >>>>>> 2013-07-12 09:08:11,402 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting
log
>>> >>>>>> segment at
>>> >>>>>> 565
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling
edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending
log
>>> >>>>>> segment 565
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number
of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions
batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> >>>>>> 2013-07-12 09:09:11,441 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number
of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions
batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> And after sometime it said:
>>> >>>>>>
>>> >>>>>> 2013-07-12 11:03:19,799 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting
log
>>> >>>>>> segment at
>>> >>>>>> 795
>>> >>>>>> 2013-07-12 11:04:19,826 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 11:04:19,826 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling
edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending
log
>>> >>>>>> segment 795
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number
of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions
batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number
of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions
batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 11:04:19,829 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> >>>>>> 2013-07-12 11:04:19,829 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting
log
>>> >>>>>> segment at
>>> >>>>>> 797
>>> >>>>>> 2013-07-12 11:04:26,002 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the
configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>> 2013-07-12 11:04:26,003 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
NameNode low
>>> >>>>>> on
>>> >>>>>> available disk space. Entering safe mode.
>>> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>> >>>>>> STATE* Safe mode is ON.
>>> >>>>>> Resources are low on NN. Please add or free up more
resources then
>>> >>>>>> turn off safe mode manually.  NOTE:  If you turn off
safe mode
>>> >>>>>> before adding
>>> >>>>>> resources, the NN will immediately return to safe mode..
>>> >>>>>> 2013-07-12 11:04:31,004 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the
configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>> 2013-07-12 11:04:31,005 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
NameNode low
>>> >>>>>> on
>>> >>>>>> available disk space. Already in safe mode.
>>> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>> >>>>>> STATE* Safe mode is ONResources are low on NN. Please
add or free
>>> >>>>>> up more
>>> >>>>>> resources then turn off safe mode manually.  NOTE: 
If you turn
>>> >>>>>> off safe
>>> >>>>>> mode before adding resources, the NN will immediately
return to
>>> >>>>>> safe mode..
>>> >>>>>> 2013-07-12 11:04:36,005 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the
configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Thanks,
>>> >>>>>> Kishore
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> N Venkata Rami Reddy
>>> >>>>> Hadoop Admin
>>> >>>>> Cloudwick Technologies
>>> >>>>
>>> >>>>
>>> >>>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>



-- 
Harsh J

Mime
View raw message