Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2C53910F37 for ; Mon, 15 Jul 2013 08:30:07 +0000 (UTC) Received: (qmail 32308 invoked by uid 500); 15 Jul 2013 08:30:00 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 31996 invoked by uid 500); 15 Jul 2013 08:29:59 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 31989 invoked by uid 99); 15 Jul 2013 08:29:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jul 2013 08:29:58 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of adityaexalter@gmail.com designates 209.85.214.169 as permitted sender) Received: from [209.85.214.169] (HELO mail-ob0-f169.google.com) (209.85.214.169) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jul 2013 08:29:52 +0000 Received: by mail-ob0-f169.google.com with SMTP id up14so13805179obb.28 for ; Mon, 15 Jul 2013 01:29:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=bDcao6SKVptrWljbFY6B5iaGt/F5QkythN+Tbh90sDM=; b=WRU1SDNxnGvMhbR83G58FgYyy2hrJrPB0O2R2lE4SRRDCITZHTtxjn7MuGiay6LpJJ bf1LZZ/TWX5HDgeiEt9IITWm9SOLglWhIkrKxjMD1piBoOyaslq/XPCwkMoge7tOSNLt ZcOMggTuSnRswEvJNUoNZJF9gU78B3R2oCKw4fWiBOwTZD0LKoSTTiDzukOmpndzKi40 oA4NtdoLOX+3X+7ZdjfSntGfXw4mCvtxEX0Z0OrR3Ba9H6HayDNgIB4B0X6c3rlz9rwD 0yu/UYltraiO/uyldh6pWD80olyNqAxZCZeLH9NUWkKvWRH8BWks2pya1sV84oUv7ppo 7PHA== MIME-Version: 1.0 X-Received: by 10.60.41.37 with SMTP id c5mr42410906oel.43.1373876971264; Mon, 15 Jul 2013 01:29:31 -0700 (PDT) Received: by 10.182.110.106 with HTTP; Mon, 15 Jul 2013 01:29:31 -0700 (PDT) In-Reply-To: References: Date: Mon, 15 Jul 2013 13:59:31 +0530 Message-ID: Subject: Re: Namenode automatically going to safemode with 2.1.0-beta From: Aditya exalter To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e013a0964d6603d04e188a874 X-Virus-Checked: Checked by ClamAV on apache.org --089e013a0964d6603d04e188a874 Content-Type: text/plain; charset=ISO-8859-1 Hi Krishna, Can you please send screenshots of namenode web UI. Thanks Aditya. On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri < write2kishore@gmail.com> wrote: > I have had enough space on the disk that is used, like around 30 Gigs > > Thanks, > Kishore > > > On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla < > venkatarami.netla@cloudwick.com> wrote: > >> Hi, >> pls see the available space for NN storage directory. >> >> Thanks & Regards >> >> Venkat >> >> >> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri < >> write2kishore@gmail.com> wrote: >> >>> Hi, >>> >>> I am doing no activity on my single node cluster which is using >>> 2.1.0-beta, and still observed that it has gone to safe mode by itself >>> after a while. I was looking at the name node log and see many of these >>> kinds of entries.. Can anything be interpreted from these? >>> >>> 2013-07-12 09:06:11,256 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>> 561 >>> 2013-07-12 09:07:11,290 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>> 9.70.137.114 >>> 2013-07-12 09:07:11,290 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>> 2013-07-12 09:07:11,290 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561 >>> 2013-07-12 09:07:11,291 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>> Total time for transactions(ms): 1 Number of transactions batched in Syncs: >>> 0 Number of syncs: 2 SyncTimes(ms): 14 >>> 2013-07-12 09:07:11,292 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>> Total time for transactions(ms): 1 Number of transactions batched in Syncs: >>> 0 Number of syncs: 3 SyncTimes(ms): 15 >>> 2013-07-12 09:07:11,293 INFO >>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits >>> file >>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 -> >>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562 >>> 2013-07-12 09:07:11,294 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>> 563 >>> 2013-07-12 09:08:11,397 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>> 9.70.137.114 >>> 2013-07-12 09:08:11,398 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>> 2013-07-12 09:08:11,398 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563 >>> 2013-07-12 09:08:11,399 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>> Total time for transactions(ms): 2 Number of transactions batched in Syncs: >>> 0 Number of syncs: 2 SyncTimes(ms): 11 >>> 2013-07-12 09:08:11,400 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>> Total time for transactions(ms): 2 Number of transactions batched in Syncs: >>> 0 Number of syncs: 3 SyncTimes(ms): 12 >>> 2013-07-12 09:08:11,402 INFO >>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits >>> file >>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 -> >>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564 >>> 2013-07-12 09:08:11,402 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>> 565 >>> 2013-07-12 09:09:11,440 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>> 9.70.137.114 >>> 2013-07-12 09:09:11,440 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>> 2013-07-12 09:09:11,440 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565 >>> 2013-07-12 09:09:11,440 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>> 0 Number of syncs: 2 SyncTimes(ms): 13 >>> 2013-07-12 09:09:11,441 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>> 0 Number of syncs: 3 SyncTimes(ms): 13 >>> >>> >>> And after sometime it said: >>> >>> 2013-07-12 11:03:19,799 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>> 795 >>> 2013-07-12 11:04:19,826 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from >>> 9.70.137.114 >>> 2013-07-12 11:04:19,826 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs >>> 2013-07-12 11:04:19,827 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795 >>> 2013-07-12 11:04:19,827 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>> 0 Number of syncs: 2 SyncTimes(ms): 12 >>> 2013-07-12 11:04:19,827 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 >>> Total time for transactions(ms): 0 Number of transactions batched in Syncs: >>> 0 Number of syncs: 3 SyncTimes(ms): 12 >>> 2013-07-12 11:04:19,829 INFO >>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits >>> file >>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 -> >>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796 >>> 2013-07-12 11:04:19,829 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at >>> 797 >>> 2013-07-12 11:04:26,002 WARN >>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>> available on volume 'null' is 0, which is below the configured reserved >>> amount 104857600 >>> 2013-07-12 11:04:26,003 WARN >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on >>> available disk space. Entering safe mode. >>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE* >>> Safe mode is ON. >>> Resources are low on NN. Please add or free up more resources then turn >>> off safe mode manually. NOTE: If you turn off safe mode before adding >>> resources, the NN will immediately return to safe mode.. >>> 2013-07-12 11:04:31,004 WARN >>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>> available on volume 'null' is 0, which is below the configured reserved >>> amount 104857600 >>> 2013-07-12 11:04:31,005 WARN >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on >>> available disk space. Already in safe mode. >>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE* >>> Safe mode is ONResources are low on NN. Please add or free up more >>> resources then turn off safe mode manually. NOTE: If you turn off safe >>> mode before adding resources, the NN will immediately return to safe mode.. >>> 2013-07-12 11:04:36,005 WARN >>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>> available on volume 'null' is 0, which is below the configured reserved >>> amount 104857600 >>> >>> >>> Thanks, >>> Kishore >>> >> >> >> >> -- >> N Venkata Rami Reddy >> Hadoop Admin >> Cloudwick Technologies >> > > --089e013a0964d6603d04e188a874 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Krishna,

=A0 =A0Can you please= send screenshots of namenode web UI.

= Thanks Aditya.


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <write2kis= hore@gmail.com> wrote:
I have had enough space on the disk that is used, like aro= und 30 Gigs=A0

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <venkatar= ami.netla@cloudwick.com> wrote:
Hi,
pls see the availab= le space for NN storage directory.

Thanks & Re= gards

Venkat
=

On Mon, Jul 15, 2013 at 12:14 PM, Krishna Ki= shore Bonagiri <write2kishore@gmail.com> wrote:
Hi,

=A0I am doing no activity on my single no= de cluster which is using 2.1.0-beta, and still observed that it has gone t= o safe mode by itself after a while. I was looking at the name node log and= see many of these kinds of entries.. Can anything be interpreted from thes= e?

2013-07-12 09:06:11,256 INFO org.apache.hadoop.hdf= s.server.namenode.FSEditLog: Starting log segment at 561
2013-07-= 12 09:07:11,290 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: R= oll Edit Log from 9.70.137.114
2013-07-12 09:07:11,290 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Rolling edit logs
2013-07-12 09:07:11,290 INFO org.apach= e.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
2013-07-12 09:07:11,291 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Number of transactions: 2 Total time for transactions(ms): 1 Number of = transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 14
2013-07-12 09:07:11,292 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 1 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 15<= /div>
2013-07-12 09:07:11,293 INFO org.apache.hadoop.hdfs.server.namenode.Fi= leJournalManager: Finalizing edits file /tmp/hadoop-dsadm/dfs/name/current/= edits_inprogress_0000000000000000561 -> /tmp/hadoop-dsadm/dfs/name/curre= nt/edits_0000000000000000561-0000000000000000562
2013-07-12 09:07:11,294 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Starting log segment at 563
2013-07-12 09:08:11,397 INFO= org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 9.= 70.137.114
2013-07-12 09:08:11,398 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Rolling edit logs
2013-07-12 09:08:11,398 INFO org.apach= e.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
2013-07-12 09:08:11,399 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Number of transactions: 2 Total time for transactions(ms): 2 Number of = transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 11
2013-07-12 09:08:11,400 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 2 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 12<= /div>
2013-07-12 09:08:11,402 INFO org.apache.hadoop.hdfs.server.namenode.Fi= leJournalManager: Finalizing edits file /tmp/hadoop-dsadm/dfs/name/current/= edits_inprogress_0000000000000000563 -> /tmp/hadoop-dsadm/dfs/name/curre= nt/edits_0000000000000000563-0000000000000000564
2013-07-12 09:08:11,402 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Starting log segment at 565
2013-07-12 09:09:11,440 INFO= org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 9.= 70.137.114
2013-07-12 09:09:11,440 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Rolling edit logs
2013-07-12 09:09:11,440 INFO org.apach= e.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
2013-07-12 09:09:11,440 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Number of transactions: 2 Total time for transactions(ms): 0 Number of = transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 13
2013-07-12 09:09:11,441 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 0 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 13<= /div>


And after sometime it said:
2013-07-12 11:03:19,799 INFO org.apache.hadoop.hdfs.serve= r.namenode.FSEditLog: Starting log segment at 795
2013-07-12 11:04:19,826 INFO org.apache.hadoop.hdfs.server.namenode.FS= Namesystem: Roll Edit Log from 9.70.137.114
2013-07-12 11:04:19,8= 26 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs=
2013-07-12 11:04:19,827 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Ending log segment 795
2013-07-12 11:04:19,827 INFO org.= apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Tot= al time for transactions(ms): 0 Number of transactions batched in Syncs: 0 = Number of syncs: 2 SyncTimes(ms): 12
2013-07-12 11:04:19,827 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Number of transactions: 2 Total time for transactions(ms): 0 Numbe= r of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 12<= /div>
2013-07-12 11:04:19,829 INFO org.apache.hadoop.hdfs.server.namenode.Fi= leJournalManager: Finalizing edits file /tmp/hadoop-dsadm/dfs/name/current/= edits_inprogress_0000000000000000795 -> /tmp/hadoop-dsadm/dfs/name/curre= nt/edits_0000000000000000795-0000000000000000796
2013-07-12 11:04:19,829 INFO org.apache.hadoop.hdfs.server.namenode.FS= EditLog: Starting log segment at 797
2013-07-12 11:04:26,002 WARN= org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space avai= lable on volume 'null' is 0, which is below the configured reserved= amount 104857600
2013-07-12 11:04:26,003 WARN org.apache.hadoop.hdfs.server.namenode.FS= Namesystem: NameNode low on available disk space. Entering safe mode.
=
2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE= * Safe mode is ON.
Resources are low on NN. Please add or free up more resources then tur= n off safe mode manually. =A0NOTE: =A0If you turn off safe mode before addi= ng resources, the NN will immediately return to safe mode..
2013-= 07-12 11:04:31,004 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeReso= urceChecker: Space available on volume 'null' is 0, which is below = the configured reserved amount 104857600
2013-07-12 11:04:31,005 WARN org.apache.hadoop.hdfs.server.namenode.FS= Namesystem: NameNode low on available disk space. Already in safe mode.
2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STA= TE* Safe mode is ONResources are low on NN. Please add or free up more reso= urces then turn off safe mode manually. =A0NOTE: =A0If you turn off safe mo= de before adding resources, the NN will immediately return to safe mode..
2013-07-12 11:04:36,005 WARN org.apache.hadoop.hdfs.server.namenode.Na= meNodeResourceChecker: Space available on volume 'null' is 0, which= is below the configured reserved amount 104857600


Thanks,
Kishore



<= font color=3D"#888888">--
N Venkata Rami Red= dy
Hadoop Admin
Cloudwick Technologies


--089e013a0964d6603d04e188a874--