Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 701CADF24 for ; Sun, 9 Sep 2012 22:39:47 +0000 (UTC) Received: (qmail 99492 invoked by uid 500); 9 Sep 2012 22:39:42 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 99303 invoked by uid 500); 9 Sep 2012 22:39:42 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 99294 invoked by uid 99); 9 Sep 2012 22:39:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Sep 2012 22:39:42 +0000 X-ASF-Spam-Status: No, hits=0.0 required=5.0 tests=FSL_RCVD_USER,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kwiley@keithwiley.com designates 69.56.183.8 as permitted sender) Received: from [69.56.183.8] (HELO gateway16.websitewelcome.com) (69.56.183.8) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Sep 2012 22:39:34 +0000 Received: by gateway16.websitewelcome.com (Postfix, from userid 5007) id BBFEB26F2C79; Sun, 9 Sep 2012 17:39:10 -0500 (CDT) Received: from gator542.hostgator.com (gator542.hostgator.com [74.54.187.114]) by gateway16.websitewelcome.com (Postfix) with ESMTP id 899AC26F2C22 for ; Sun, 9 Sep 2012 17:39:10 -0500 (CDT) Received: from [24.19.6.8] (port=47965 helo=[192.168.10.2]) by gator542.hostgator.com with esmtpa (Exim 4.77) (envelope-from ) id 1TAq9o-0005yT-Hz for user@hadoop.apache.org; Sun, 09 Sep 2012 17:39:12 -0500 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Apple Message framework v1084) Subject: Re: Can't get out of safemode From: Keith Wiley In-Reply-To: Date: Sun, 9 Sep 2012 15:39:12 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <9E24E8AD-5B06-4642-A5AB-DA0E2F497246@keithwiley.com> References: <6D9B037A-796C-4CF4-854E-95EB4BDC2CD6@keithwiley.com> To: user@hadoop.apache.org X-Mailer: Apple Mail (2.1084) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - gator542.hostgator.com X-AntiAbuse: Original Domain - hadoop.apache.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - keithwiley.com X-BWhitelist: no X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: ([192.168.10.2]) [24.19.6.8]:47965 X-Source-Auth: kwiley+keithwiley.com X-Email-Count: 2 X-Source-Cap: a2J3aWxleTtrYndpbGV5O2dhdG9yNTQyLmhvc3RnYXRvci5jb20= Yep, moving the NN to the secondary larger disk seems to have improved = it. Thanks again for your great help Harsh (and others). Cheers! On Sep 7, 2012, at 08:31 , Harsh J wrote: > The issue was that Keith's NN was out of resources (low disk space) > and that forced the NN into safemode. This was detected on another > thread from him. >=20 > On Fri, Sep 7, 2012 at 1:12 PM, Dino Ke=C4=8Do = wrote: >> Hi Keith, >>=20 >> I was having same problem on my small cluster and some blocks where = missing >> on DFS. I have used this command >>=20 >> hadoop dfsadmin -safemode leave >>=20 >> and after that i have run dfs disk check to see which files are = corrupted. >>=20 >> Hope this helps, >> Dino Ke=C4=8Do >> msn: xdinno@hotmail.com >> mail: dino.keco@gmail.com >> skype: dino.keco >> phone: +387 61 507 851 >>=20 >>=20 >> On Fri, Sep 7, 2012 at 12:13 AM, Adam Brown = wrote: >>>=20 >>> sorry >>>=20 >>> @Keith >>>=20 >>> are you sure your datanodes have reported in? >>>=20 >>> On Thu, Sep 6, 2012 at 3:12 PM, Adam Brown = wrote: >>>> Hi Serge, >>>>=20 >>>> are you sure your datanodes have reported in ? >>>>=20 >>>>=20 >>>>=20 >>>> On Tue, Sep 4, 2012 at 10:10 AM, Serge Blazhiyevskyy >>>> wrote: >>>>> Can look in name node logs and post last few lines? >>>>>=20 >>>>>=20 >>>>>=20 >>>>> On 9/4/12 10:07 AM, "Keith Wiley" wrote: >>>>>=20 >>>>>> Observe: >>>>>>=20 >>>>>> ~/ $ hd fs -put test /test >>>>>> put: org.apache.hadoop.hdfs.server.namenode.SafeModeException: = Cannot >>>>>> create file/test. Name node is in safe mode. >>>>>> ~/ $ hadoop dfsadmin -safemode leave >>>>>> Safe mode is OFF >>>>>> ~/ $ hadoop dfsadmin -safemode get >>>>>> Safe mode is ON >>>>>> ~/ $ hadoop dfsadmin -safemode leave >>>>>> Safe mode is OFF >>>>>> ~/ $ hadoop dfsadmin -safemode get >>>>>> Safe mode is ON >>>>>> ~/ $ hd fs -put test /test >>>>>> put: org.apache.hadoop.hdfs.server.namenode.SafeModeException: = Cannot >>>>>> create file/test. Name node is in safe mode. >>>>>> ~/ $ >>>>>>=20 >>>>>> Grrr! = __________________________________________________________________________= ______ Keith Wiley kwiley@keithwiley.com keithwiley.com = music.keithwiley.com "Yet mark his perfect self-contentment, and hence learn his lesson, that = to be self-contented is to be vile and ignorant, and that to aspire is better = than to be blindly and impotently happy." -- Edwin A. Abbott, Flatland = __________________________________________________________________________= ______