Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7D9DD10FE4 for ; Mon, 15 Jul 2013 08:59:51 +0000 (UTC) Received: (qmail 77288 invoked by uid 500); 15 Jul 2013 08:59:43 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 77152 invoked by uid 500); 15 Jul 2013 08:59:41 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 77143 invoked by uid 99); 15 Jul 2013 08:59:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jul 2013 08:59:40 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of azuryyyu@gmail.com designates 209.85.212.42 as permitted sender) Received: from [209.85.212.42] (HELO mail-vb0-f42.google.com) (209.85.212.42) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jul 2013 08:59:35 +0000 Received: by mail-vb0-f42.google.com with SMTP id i3so2975864vbh.29 for ; Mon, 15 Jul 2013 01:59:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=ZfbizYKLN8c/nIbsvU5OHto5huGN9QbfV8uwmQHSD6s=; b=cGMlcnYLo5g/WoeulIEFvRLXA4tyHh6txmdafevfj/8cRfyVDshUfdFwMewM4AEknW KLNa9SQC/R6gn7CG9JFA38HFSCdVG5sPdcBE1Xxf6x5kSC8nqLsTfoQFkHvR9hvVtx41 6bKj17FN2p5UlUCPiUtp0/Y7EnDOQ6FMU1AblsG2vXceQFeO2hP5nU/rMnB03KX8xX7c rjLfhNwt/J0PEwnQLvwrKQ7RjiEHCE1Rj3y7kzlWtAUeTsW1N1aIuTYcucJerUzBRPXX 7evhdywXvvQMUAqagNIj/UQzR/J1+SD5Yli6R60EBY6qvnDeuwJ1fhvjy8Y2l1vpOcgm 8JpA== MIME-Version: 1.0 X-Received: by 10.58.19.162 with SMTP id g2mr28346103vee.12.1373878754804; Mon, 15 Jul 2013 01:59:14 -0700 (PDT) Received: by 10.220.15.6 with HTTP; Mon, 15 Jul 2013 01:59:14 -0700 (PDT) Received: by 10.220.15.6 with HTTP; Mon, 15 Jul 2013 01:59:14 -0700 (PDT) In-Reply-To: References: Date: Mon, 15 Jul 2013 16:59:14 +0800 Message-ID: Subject: Re: Namenode automatically going to safemode with 2.1.0-beta From: Azuryy Yu To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b86e63a250b2f04e18913a2 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b86e63a250b2f04e18913a2 Content-Type: text/plain; charset=ISO-8859-1 please check dfs.datanode.du.reserved in the hdfs-site.xml On Jul 15, 2013 4:30 PM, "Aditya exalter" wrote: > Hi Krishna, > > Can you please send screenshots of namenode web UI. > > Thanks Aditya. > > > On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri < > write2kishore@gmail.com> wrote: > >> I have had enough space on the disk that is used, like around 30 Gigs >> >> Thanks, >> Kishore >> >> >> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla < >> venkatarami.netla@cloudwick.com> wrote: >> >>> Hi, >>> pls see the available space for NN storage directory. >>> >>> Thanks & Regards >>> >>> Venkat >>> >>> >>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri < >>> write2kishore@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I am doing no activity on my single node cluster which is using >>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself >>>> after a while. I was looking at the name node log and see many of these >>>> kinds of entries.. Can anything be interpreted from these? >>>> >>>> 2013-07-12 09:06:11,256 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>>> 561 >>>> 2013-07-12 09:07:11,290 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>>> 9.70.137.114 >>>> 2013-07-12 09:07:11,290 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>>> 2013-07-12 09:07:11,290 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561 >>>> 2013-07-12 09:07:11,291 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs: >>>> 0 Number of syncs: 2 SyncTimes(ms): 14 >>>> 2013-07-12 09:07:11,292 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs: >>>> 0 Number of syncs: 3 SyncTimes(ms): 15 >>>> 2013-07-12 09:07:11,293 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits >>>> file >>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 -> >>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562 >>>> 2013-07-12 09:07:11,294 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>>> 563 >>>> 2013-07-12 09:08:11,397 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>>> 9.70.137.114 >>>> 2013-07-12 09:08:11,398 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>>> 2013-07-12 09:08:11,398 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563 >>>> 2013-07-12 09:08:11,399 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs: >>>> 0 Number of syncs: 2 SyncTimes(ms): 11 >>>> 2013-07-12 09:08:11,400 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs: >>>> 0 Number of syncs: 3 SyncTimes(ms): 12 >>>> 2013-07-12 09:08:11,402 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits >>>> file >>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 -> >>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564 >>>> 2013-07-12 09:08:11,402 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>>> 565 >>>> 2013-07-12 09:09:11,440 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>>> 9.70.137.114 >>>> 2013-07-12 09:09:11,440 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>>> 2013-07-12 09:09:11,440 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565 >>>> 2013-07-12 09:09:11,440 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>>> 0 Number of syncs: 2 SyncTimes(ms): 13 >>>> 2013-07-12 09:09:11,441 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>>> 0 Number of syncs: 3 SyncTimes(ms): 13 >>>> >>>> >>>> And after sometime it said: >>>> >>>> 2013-07-12 11:03:19,799 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>>> 795 >>>> 2013-07-12 11:04:19,826 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>>> 9.70.137.114 >>>> 2013-07-12 11:04:19,826 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>>> 2013-07-12 11:04:19,827 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795 >>>> 2013-07-12 11:04:19,827 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>>> 0 Number of syncs: 2 SyncTimes(ms): 12 >>>> 2013-07-12 11:04:19,827 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>>> 0 Number of syncs: 3 SyncTimes(ms): 12 >>>> 2013-07-12 11:04:19,829 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits >>>> file >>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 -> >>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796 >>>> 2013-07-12 11:04:19,829 INFO >>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>>> 797 >>>> 2013-07-12 11:04:26,002 WARN >>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>>> available on volume 'null' is 0, which is below the configured reserved >>>> amount 104857600 >>>> 2013-07-12 11:04:26,003 WARN >>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on >>>> available disk space. Entering safe mode. >>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE* >>>> Safe mode is ON. >>>> Resources are low on NN. Please add or free up more resources then turn >>>> off safe mode manually. NOTE: If you turn off safe mode before adding >>>> resources, the NN will immediately return to safe mode.. >>>> 2013-07-12 11:04:31,004 WARN >>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>>> available on volume 'null' is 0, which is below the configured reserved >>>> amount 104857600 >>>> 2013-07-12 11:04:31,005 WARN >>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on >>>> available disk space. Already in safe mode. >>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE* >>>> Safe mode is ONResources are low on NN. Please add or free up more >>>> resources then turn off safe mode manually. NOTE: If you turn off safe >>>> mode before adding resources, the NN will immediately return to safe mode.. >>>> 2013-07-12 11:04:36,005 WARN >>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>>> available on volume 'null' is 0, which is below the configured reserved >>>> amount 104857600 >>>> >>>> >>>> Thanks, >>>> Kishore >>>> >>> >>> >>> >>> -- >>> N Venkata Rami Reddy >>> Hadoop Admin >>> Cloudwick Technologies >>> >> >> > --047d7b86e63a250b2f04e18913a2 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

please check dfs.datanode.du.reserved in the hdfs-site.xml

On Jul 15, 2013 4:30 PM, "Aditya exalter&qu= ot; <adityaexalter@gmail.com<= /a>> wrote:
Hi Krishna,

=A0 =A0Can you please send = screenshots of namenode web UI.

Thanks Aditya.


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <write2kis= hore@gmail.com> wrote:
I have had enough space on the disk that is used, like aro= und 30 Gigs=A0

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <venkatar= ami.netla@cloudwick.com> wrote:
Hi,
pls see the availab= le space for NN storage directory.

Thanks & Re= gards

Venkat
=

On Mon, Jul 15, 2013 at 12:14 PM, Krishna Ki= shore Bonagiri <write2kishore@gmail.com> wrote:
Hi,

=A0I am doing no activity on my single no= de cluster which is using 2.1.0-beta, and still observed that it has gone t= o safe mode by itself after a while. I was looking at the name node log and= see many of these kinds of entries.. Can anything be interpreted from thes= e?

2013-07-12 09:06:11,256 INFO org.apache.hadoop.hdf= s.server.namenode.FSEditLog: Starting log segment at 561
2013-07-= 12 09:07:11,290 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: R= oll Edit Log from 9.70.137.114
2013-07-12 09:07:11,290 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Rolling edit logs
2013-07-12 09:07:11,290 INFO org.apach= e.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
2013-07-12 09:07:11,291 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Number of transactions: 2 Total time for transactions(ms): 1 Number of = transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 14
2013-07-12 09:07:11,292 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 1 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 15<= /div>
2013-07-12 09:07:11,293 INFO org.apache.hadoop.hdfs.server.namenode.Fi= leJournalManager: Finalizing edits file /tmp/hadoop-dsadm/dfs/name/current/= edits_inprogress_0000000000000000561 -> /tmp/hadoop-dsadm/dfs/name/curre= nt/edits_0000000000000000561-0000000000000000562
2013-07-12 09:07:11,294 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Starting log segment at 563
2013-07-12 09:08:11,397 INFO= org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 9.= 70.137.114
2013-07-12 09:08:11,398 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Rolling edit logs
2013-07-12 09:08:11,398 INFO org.apach= e.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
2013-07-12 09:08:11,399 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Number of transactions: 2 Total time for transactions(ms): 2 Number of = transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 11
2013-07-12 09:08:11,400 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 2 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 12<= /div>
2013-07-12 09:08:11,402 INFO org.apache.hadoop.hdfs.server.namenode.Fi= leJournalManager: Finalizing edits file /tmp/hadoop-dsadm/dfs/name/current/= edits_inprogress_0000000000000000563 -> /tmp/hadoop-dsadm/dfs/name/curre= nt/edits_0000000000000000563-0000000000000000564
2013-07-12 09:08:11,402 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Starting log segment at 565
2013-07-12 09:09:11,440 INFO= org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 9.= 70.137.114
2013-07-12 09:09:11,440 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Rolling edit logs
2013-07-12 09:09:11,440 INFO org.apach= e.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
2013-07-12 09:09:11,440 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Number of transactions: 2 Total time for transactions(ms): 0 Number of = transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 13
2013-07-12 09:09:11,441 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 0 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 13<= /div>


And after sometime it said:
2013-07-12 11:03:19,799 INFO org.apache.hadoop.hdfs.serve= r.namenode.FSEditLog: Starting log segment at 795
2013-07-12 11:04:19,826 INFO org.apache.hadoop.hdfs.server.namenode.FS= Namesystem: Roll Edit Log from 9.70.137.114
2013-07-12 11:04:19,8= 26 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs=
2013-07-12 11:04:19,827 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Ending log segment 795
2013-07-12 11:04:19,827 INFO org.= apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Tot= al time for transactions(ms): 0 Number of transactions batched in Syncs: 0 = Number of syncs: 2 SyncTimes(ms): 12
2013-07-12 11:04:19,827 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 0 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 12<= /div>
2013-07-12 11:04:19,829 INFO org.apache.hadoop.hdfs.server.namenode.Fi= leJournalManager: Finalizing edits file /tmp/hadoop-dsadm/dfs/name/current/= edits_inprogress_0000000000000000795 -> /tmp/hadoop-dsadm/dfs/name/curre= nt/edits_0000000000000000795-0000000000000000796
2013-07-12 11:04:19,829 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Starting log segment at 797
2013-07-12 11:04:26,002 WARN= org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space avai= lable on volume 'null' is 0, which is below the configured reserved= amount 104857600
2013-07-12 11:04:26,003 WARN org.apache.hadoop.hdfs.server.namenode.FS= Namesystem: NameNode low on available disk space. Entering safe mode.
=
2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE= * Safe mode is ON.
Resources are low on NN. Please add or free up more resources then tur= n off safe mode manually. =A0NOTE: =A0If you turn off safe mode before addi= ng resources, the NN will immediately return to safe mode..
2013-= 07-12 11:04:31,004 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeReso= urceChecker: Space available on volume 'null' is 0, which is below = the configured reserved amount 104857600
2013-07-12 11:04:31,005 WARN org.apache.hadoop.hdfs.server.namenode.FS= Namesystem: NameNode low on available disk space. Already in safe mode.
2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STA= TE* Safe mode is ONResources are low on NN. Please add or free up more reso= urces then turn off safe mode manually. =A0NOTE: =A0If you turn off safe mo= de before adding resources, the NN will immediately return to safe mode..
2013-07-12 11:04:36,005 WARN org.apache.hadoop.hdfs.server.namenode.Na= meNodeResourceChecker: Space available on volume 'null' is 0, which= is below the configured reserved amount 104857600


Thanks,
Kishore



<= font color=3D"#888888">--
N Venkata Rami Red= dy
Hadoop Admin
Cloudwick Technologies


--047d7b86e63a250b2f04e18913a2--