Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 451C510094 for ; Mon, 15 Jul 2013 09:14:03 +0000 (UTC) Received: (qmail 10585 invoked by uid 500); 15 Jul 2013 09:13:57 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 10418 invoked by uid 500); 15 Jul 2013 09:13:56 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 10411 invoked by uid 99); 15 Jul 2013 09:13:56 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jul 2013 09:13:56 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of write2kishore@gmail.com designates 209.85.223.181 as permitted sender) Received: from [209.85.223.181] (HELO mail-ie0-f181.google.com) (209.85.223.181) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jul 2013 09:13:50 +0000 Received: by mail-ie0-f181.google.com with SMTP id x12so25597611ief.12 for ; Mon, 15 Jul 2013 02:13:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=lbjFx+O8VYLRUIyWMWdqBrHYas0V/iwDMgBRCsHCwWI=; b=wApc6BLvuEgu388INk46F9ZnEpUACoqGQRpy83VJYJ1F9PVbV5SKvqyMYe5jak4aPQ iiKD8/Tucj/oUUVnQKGcdaikBIAD5zW7w21NGy/VngZkB/mGQN4pUFtbJvKY1FOZ4EJf wwtyCWl7k0Wn8SvGTTjL21Ry38HNxe7E/x047O8esjMpqCW+VWkibWBIqP/cjz6pjhwi fCZ89j192xIGj4pMH3uEYtpsBFQZZasswBlCvisXzzpru3H7JFXeJbBPAIkBpTj8PEeh gfyqsOqZEglAVB84oVD8/5uSJ7AqA6pp7UVfq5zwrgc4VkpIPxGU5Wd4BKm3Jyn4I6cW XkMg== MIME-Version: 1.0 X-Received: by 10.50.22.72 with SMTP id b8mr6515442igf.17.1373879609191; Mon, 15 Jul 2013 02:13:29 -0700 (PDT) Received: by 10.42.66.13 with HTTP; Mon, 15 Jul 2013 02:13:29 -0700 (PDT) In-Reply-To: References: Date: Mon, 15 Jul 2013 14:43:29 +0530 Message-ID: Subject: Re: Namenode automatically going to safemode with 2.1.0-beta From: Krishna Kishore Bonagiri To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b10c94711f35d04e1894605 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b10c94711f35d04e1894605 Content-Type: text/plain; charset=ISO-8859-1 I don't have it in my hdfs-site.xml, in which case probably the default value is taken.. On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu wrote: > please check dfs.datanode.du.reserved in the hdfs-site.xml > On Jul 15, 2013 4:30 PM, "Aditya exalter" wrote: > >> Hi Krishna, >> >> Can you please send screenshots of namenode web UI. >> >> Thanks Aditya. >> >> >> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri < >> write2kishore@gmail.com> wrote: >> >>> I have had enough space on the disk that is used, like around 30 Gigs >>> >>> Thanks, >>> Kishore >>> >>> >>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla < >>> venkatarami.netla@cloudwick.com> wrote: >>> >>>> Hi, >>>> pls see the available space for NN storage directory. >>>> >>>> Thanks & Regards >>>> >>>> Venkat >>>> >>>> >>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri < >>>> write2kishore@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am doing no activity on my single node cluster which is using >>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself >>>>> after a while. I was looking at the name node log and see many of these >>>>> kinds of entries.. Can anything be interpreted from these? >>>>> >>>>> 2013-07-12 09:06:11,256 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>>>> 561 >>>>> 2013-07-12 09:07:11,290 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>>>> 9.70.137.114 >>>>> 2013-07-12 09:07:11,290 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>>>> 2013-07-12 09:07:11,290 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561 >>>>> 2013-07-12 09:07:11,291 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs: >>>>> 0 Number of syncs: 2 SyncTimes(ms): 14 >>>>> 2013-07-12 09:07:11,292 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs: >>>>> 0 Number of syncs: 3 SyncTimes(ms): 15 >>>>> 2013-07-12 09:07:11,293 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits >>>>> file >>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 -> >>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562 >>>>> 2013-07-12 09:07:11,294 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>>>> 563 >>>>> 2013-07-12 09:08:11,397 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>>>> 9.70.137.114 >>>>> 2013-07-12 09:08:11,398 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>>>> 2013-07-12 09:08:11,398 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563 >>>>> 2013-07-12 09:08:11,399 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs: >>>>> 0 Number of syncs: 2 SyncTimes(ms): 11 >>>>> 2013-07-12 09:08:11,400 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs: >>>>> 0 Number of syncs: 3 SyncTimes(ms): 12 >>>>> 2013-07-12 09:08:11,402 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits >>>>> file >>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 -> >>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564 >>>>> 2013-07-12 09:08:11,402 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>>>> 565 >>>>> 2013-07-12 09:09:11,440 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>>>> 9.70.137.114 >>>>> 2013-07-12 09:09:11,440 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>>>> 2013-07-12 09:09:11,440 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565 >>>>> 2013-07-12 09:09:11,440 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>>>> 0 Number of syncs: 2 SyncTimes(ms): 13 >>>>> 2013-07-12 09:09:11,441 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>>>> 0 Number of syncs: 3 SyncTimes(ms): 13 >>>>> >>>>> >>>>> And after sometime it said: >>>>> >>>>> 2013-07-12 11:03:19,799 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>>>> 795 >>>>> 2013-07-12 11:04:19,826 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>>>> 9.70.137.114 >>>>> 2013-07-12 11:04:19,826 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>>>> 2013-07-12 11:04:19,827 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795 >>>>> 2013-07-12 11:04:19,827 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>>>> 0 Number of syncs: 2 SyncTimes(ms): 12 >>>>> 2013-07-12 11:04:19,827 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>>>> 0 Number of syncs: 3 SyncTimes(ms): 12 >>>>> 2013-07-12 11:04:19,829 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits >>>>> file >>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 -> >>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796 >>>>> 2013-07-12 11:04:19,829 INFO >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>>>> 797 >>>>> 2013-07-12 11:04:26,002 WARN >>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>>>> available on volume 'null' is 0, which is below the configured reserved >>>>> amount 104857600 >>>>> 2013-07-12 11:04:26,003 WARN >>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on >>>>> available disk space. Entering safe mode. >>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: >>>>> STATE* Safe mode is ON. >>>>> Resources are low on NN. Please add or free up more resources then >>>>> turn off safe mode manually. NOTE: If you turn off safe mode before >>>>> adding resources, the NN will immediately return to safe mode.. >>>>> 2013-07-12 11:04:31,004 WARN >>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>>>> available on volume 'null' is 0, which is below the configured reserved >>>>> amount 104857600 >>>>> 2013-07-12 11:04:31,005 WARN >>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on >>>>> available disk space. Already in safe mode. >>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: >>>>> STATE* Safe mode is ONResources are low on NN. Please add or free up more >>>>> resources then turn off safe mode manually. NOTE: If you turn off safe >>>>> mode before adding resources, the NN will immediately return to safe mode.. >>>>> 2013-07-12 11:04:36,005 WARN >>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>>>> available on volume 'null' is 0, which is below the configured reserved >>>>> amount 104857600 >>>>> >>>>> >>>>> Thanks, >>>>> Kishore >>>>> >>>> >>>> >>>> >>>> -- >>>> N Venkata Rami Reddy >>>> Hadoop Admin >>>> Cloudwick Technologies >>>> >>> >>> >> --047d7b10c94711f35d04e1894605 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I don't have it in my hdfs-site.xml, in which case pro= bably the default value is taken..


=
On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az= uryyyu@gmail.com> wrote:

please check dfs.datanode.du.= reserved in the hdfs-site.xml

On Jul 15, 2013 4:30 PM, "Aditya exalter&qu= ot; <aditya= exalter@gmail.com> wrote:
Hi Krishna,

=A0 =A0Can you please send = screenshots of namenode web UI.

Thanks Aditya.


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <write2kis= hore@gmail.com> wrote:
I have had enough space on the disk that is used, like aro= und 30 Gigs=A0

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <venkatar= ami.netla@cloudwick.com> wrote:
Hi,
pls see the availab= le space for NN storage directory.

Thanks & Re= gards

Venkat
=

On Mon, Jul 15, 2013 at 12:14 PM, Krishna Ki= shore Bonagiri <write2kishore@gmail.com> wrote:
Hi,

=A0I am doing no activity on my single no= de cluster which is using 2.1.0-beta, and still observed that it has gone t= o safe mode by itself after a while. I was looking at the name node log and= see many of these kinds of entries.. Can anything be interpreted from thes= e?

2013-07-12 09:06:11,256 INFO org.apache.hadoop.hdf= s.server.namenode.FSEditLog: Starting log segment at 561
2013-07-= 12 09:07:11,290 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: R= oll Edit Log from 9.70.137.114
2013-07-12 09:07:11,290 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Rolling edit logs
2013-07-12 09:07:11,290 INFO org.apach= e.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
2013-07-12 09:07:11,291 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Number of transactions: 2 Total time for transactions(ms): 1 Number of = transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 14
2013-07-12 09:07:11,292 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 1 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 15<= /div>
2013-07-12 09:07:11,293 INFO org.apache.hadoop.hdfs.server.namenode.Fi= leJournalManager: Finalizing edits file /tmp/hadoop-dsadm/dfs/name/current/= edits_inprogress_0000000000000000561 -> /tmp/hadoop-dsadm/dfs/name/curre= nt/edits_0000000000000000561-0000000000000000562
2013-07-12 09:07:11,294 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Starting log segment at 563
2013-07-12 09:08:11,397 INFO= org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 9.= 70.137.114
2013-07-12 09:08:11,398 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Rolling edit logs
2013-07-12 09:08:11,398 INFO org.apach= e.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
2013-07-12 09:08:11,399 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Number of transactions: 2 Total time for transactions(ms): 2 Number of = transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 11
2013-07-12 09:08:11,400 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 2 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 12<= /div>
2013-07-12 09:08:11,402 INFO org.apache.hadoop.hdfs.server.namenode.Fi= leJournalManager: Finalizing edits file /tmp/hadoop-dsadm/dfs/name/current/= edits_inprogress_0000000000000000563 -> /tmp/hadoop-dsadm/dfs/name/curre= nt/edits_0000000000000000563-0000000000000000564
2013-07-12 09:08:11,402 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Starting log segment at 565
2013-07-12 09:09:11,440 INFO= org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 9.= 70.137.114
2013-07-12 09:09:11,440 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Rolling edit logs
2013-07-12 09:09:11,440 INFO org.apach= e.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
2013-07-12 09:09:11,440 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Number of transactions: 2 Total time for transactions(ms): 0 Number of = transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 13
2013-07-12 09:09:11,441 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 0 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 13<= /div>


And after sometime it said:
2013-07-12 11:03:19,799 INFO org.apache.hadoop.hdfs.serve= r.namenode.FSEditLog: Starting log segment at 795
2013-07-12 11:04:19,826 INFO org.apache.hadoop.hdfs.server.namenode.FS= Namesystem: Roll Edit Log from 9.70.137.114
2013-07-12 11:04:19,8= 26 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs=
2013-07-12 11:04:19,827 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Ending log segment 795
2013-07-12 11:04:19,827 INFO org.= apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Tot= al time for transactions(ms): 0 Number of transactions batched in Syncs: 0 = Number of syncs: 2 SyncTimes(ms): 12
2013-07-12 11:04:19,827 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 0 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 12<= /div>
2013-07-12 11:04:19,829 INFO org.apache.hadoop.hdfs.server.namenode.Fi= leJournalManager: Finalizing edits file /tmp/hadoop-dsadm/dfs/name/current/= edits_inprogress_0000000000000000795 -> /tmp/hadoop-dsadm/dfs/name/curre= nt/edits_0000000000000000795-0000000000000000796
2013-07-12 11:04:19,829 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Starting log segment at 797
2013-07-12 11:04:26,002 WARN= org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space avai= lable on volume 'null' is 0, which is below the configured reserved= amount 104857600
2013-07-12 11:04:26,003 WARN org.apache.hadoop.hdfs.server.namenode.FS= Namesystem: NameNode low on available disk space. Entering safe mode.
=
2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE= * Safe mode is ON.
Resources are low on NN. Please add or free up more resources then tur= n off safe mode manually. =A0NOTE: =A0If you turn off safe mode before addi= ng resources, the NN will immediately return to safe mode..
2013-= 07-12 11:04:31,004 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeReso= urceChecker: Space available on volume 'null' is 0, which is below = the configured reserved amount 104857600
2013-07-12 11:04:31,005 WARN org.apache.hadoop.hdfs.server.namenode.FS= Namesystem: NameNode low on available disk space. Already in safe mode.
2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STA= TE* Safe mode is ONResources are low on NN. Please add or free up more reso= urces then turn off safe mode manually. =A0NOTE: =A0If you turn off safe mo= de before adding resources, the NN will immediately return to safe mode..
2013-07-12 11:04:36,005 WARN org.apache.hadoop.hdfs.server.namenode.Na= meNodeResourceChecker: Space available on volume 'null' is 0, which= is below the configured reserved amount 104857600


Thanks,
Kishore



<= font color=3D"#888888">--
N Venkata Rami Red= dy
Hadoop Admin
Cloudwick Technologies



--047d7b10c94711f35d04e1894605--