From core-user-return-11096-apmail-hadoop-core-user-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 08 23:18:07 2009 Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 15144 invoked from network); 8 Jan 2009 23:18:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 8 Jan 2009 23:18:05 -0000 Received: (qmail 89423 invoked by uid 500); 8 Jan 2009 23:17:59 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 89376 invoked by uid 500); 8 Jan 2009 23:17:59 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 89365 invoked by uid 99); 8 Jan 2009 23:17:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jan 2009 15:17:59 -0800 X-ASF-Spam-Status: No, hits=4.0 required=10.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of stefan.will@gmx.net designates 213.165.64.20 as permitted sender) Received: from [213.165.64.20] (HELO mail.gmx.net) (213.165.64.20) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 08 Jan 2009 23:17:47 +0000 Received: (qmail invoked by alias); 08 Jan 2009 23:17:21 -0000 Received: from unknown (EHLO [192.168.5.153]) [208.178.63.106] by mail.gmx.net (mp046) with SMTP; 09 Jan 2009 00:17:21 +0100 X-Authenticated: #2748410 X-Provags-ID: V01U2FsdGVkX197EU0VFBx7fRZ0xX5eeAt9nLzsZtI13tQM2zhclq bNuUK7TvC0XKms User-Agent: Microsoft-Entourage/12.15.0.081119 Date: Thu, 08 Jan 2009 15:17:13 -0800 Subject: Re: NameNode fatal crash - 0.18.1 From: Stefan Will To: Message-ID: Thread-Topic: NameNode fatal crash - 0.18.1 Thread-Index: Aclez8hjgvZ3NeM/Q8WjUuGKhnbFBAACiluQAAVOiSAEtZvq+AAIaFNX In-Reply-To: Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3314272641_5350170" X-Y-GMX-Trusted: 0 X-FuHaFi: 0.46,-0.07000000000000001 X-Virus-Checked: Checked by ClamAV on apache.org --B_3314272641_5350170 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable I did some more poking around the name node code to find out what might hav= e caused it to deem the storage directories to be inaccessible. Apparently it does this when it catches an IOException while trying to write to or append to the edit log. I guess that makes sense, but unfortunately it doesn't actually log the actual exception: try { =20 EditLogFileOutputStream eStream =3D new EditLogFileOutputStream(getEditNewFile(sd)); eStream.create(); editStreams.add(eStream); } catch (IOException e) { // remove stream and this storage directory from list processIOError(sd); it.remove(); } -- Stefan > From: Stefan Will > Reply-To: > Date: Thu, 08 Jan 2009 11:16:29 -0800 > To: > Subject: Re: NameNode fatal crash - 0.18.1 >=20 > Jonathan, >=20 > It looks like the same thing happened to us today (on Hadoop 0.19.0). We > were running a nightly backup, and at some point, the namenode said: >=20 > > 2009-01-08 05:57:28,021 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > commitBlockSynchronization(lastblock=3Dblk_2140680350762285267_117754, > newgenerationstamp=3D117757, newlength=3D44866560, newtargets=3D[10.1.20. > 116:50010, 10.1.20.111:50010]) > 2009-01-08 05:57:30,270 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Unable to sync edit > log. Fatal Error. > 2009-01-08 05:57:30,882 FATAL > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All > storage directories are inaccessible. > 2009-01-08 05:57:31,072 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: > > Now when I try and start up the namenode again, I get an EOFException: >=20 > > 2009-01-08 10:41:45,465 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed. > java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:180) > at java.io.DataInputStream.readLong(DataInputStream.java:399) > at=20 > org.apache.hadoop.hdfs.server.namenode.FSImage.readCheckpointTime(FSImage= .ja > va:549) > at=20 > org.apache.hadoop.hdfs.server.namenode.FSImage.getFields(FSImage.java:540= ) > at=20 > org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.read(Storag= e.j > ava:227) > at=20 > org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.read(Storag= e.j > ava:216) > at=20 > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSIm= age > .java:289) > at=20 > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirector= y.j > ava:87) > at=20 > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesyst= em. > java:311) > at=20 > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.j= ava > :290) > at=20 > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:= 163 > ) > at=20 > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:208) > at=20 > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:194) > at=20 > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.j= ava > :859) > at=20 > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868) > >=20 > Did you ever figure out why your backup caused this to happen ? Our backu= p > wasn=B9t even touching the Hadoop partitions on the master. Were you able t= o > recover your DFS state ? >=20 > -- Stefan >=20 >=20 >> From: Jonathan Gray >> Reply-To: >> Date: Mon, 15 Dec 2008 12:35:39 -0800 >> To: >> Subject: RE: NameNode fatal crash - 0.18.1 >>=20 >> I have fixed the issue with the SecondaryNameNode not contacting primary >> with the 'dfs.http.address' config option. >>=20 >> Other issues still unsolved. >>=20 >>> -----Original Message----- >>> From: Jonathan Gray [mailto:jlist@streamy.com] >>> Sent: Monday, December 15, 2008 10:55 AM >>> To: core-user@hadoop.apache.org >>> Subject: NameNode fatal crash - 0.18.1 >>>=20 >>> I have a 10+1 node cluster, each slave running >>> DataNode/TaskTracker/HBase >>> RegionServer. >>>=20 >>> At the time of this crash, NameNode and SecondaryNameNode were both >>> hosted >>> on same master. >>>=20 >>> We do a nightly backup and about 95% of the way through, HDFS crashed >>> with... >>>=20 >>> NameNode shows: >>>=20 >>> 2008-12-15 01:49:31,178 ERROR org.apache.hadoop.fs.FSNamesystem: Unable >>> to >>> sync edit log. Fatal Error. >>> 2008-12-15 01:49:31,178 FATAL org.apache.hadoop.fs.FSNamesystem: Fatal >>> Error >>> : All storage directories are inaccessible. >>> 2008-12-15 01:49:31,179 INFO org.apache.hadoop.dfs.NameNode: >>> SHUTDOWN_MSG: >>>=20 >>> Every single DataNode shows: >>>=20 >>> 2008-12-15 01:49:32,340 WARN org.apache.hadoop.dfs.DataNode: >>> java.io.IOException: Call failed on local exception >>> at org.apache.hadoop.ipc.Client.call(Client.java:718) >>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) >>> at org.apache.hadoop.dfs.$Proxy4.sendHeartbeat(Unknown Source) >>> at >>> org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:655) >>> at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2888) >>> at java.lang.Thread.run(Thread.java:636) >>> Caused by: java.io.EOFException >>> at java.io.DataInputStream.readInt(DataInputStream.java:392) >>> at >>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:499 >>> ) >>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:441) >>>=20 >>>=20 >>> This is virtually all of the information I have. At the same time as >>> the >>> backup, we have normal HBase traffic and our hourly batch MR jobs. So >>> slave >>> nodes were pretty heavily loaded, but don't see anything in DN logs >>> besides >>> this Call failed. There are no space issues or anything else, Ganglia >>> shows >>> high CPU load around this time which has been typical every night, but >>> I >>> don't see any issues in DN's or NN about expired leases/no >>> heartbeats/etc. >>>=20 >>> Is there a way to prevent this failure from happening in the first >>> place? I >>> guess just reduce total load across cluster? >>>=20 >>> Second question is about how to recover once NameNode does fail... >>>=20 >>> When trying to bring HDFS back up, we get hundreds of: >>>=20 >>> 2008-12-15 07:54:13,265 ERROR org.apache.hadoop.dfs.LeaseManager: XXX >>> not >>> found in lease.paths >>>=20 >>> And then >>>=20 >>> 2008-12-15 07:54:13,267 ERROR org.apache.hadoop.fs.FSNamesystem: >>> FSNamesystem initialization failed. >>>=20 >>>=20 >>> Is there a way to recover from this? As of time of this crash, we had >>> SecondaryNameNode on the same node. Moving it to another node with >>> sufficient memory now, but would that even prevent this kind of FS >>> botching? >>>=20 >>> Also, my SecondaryNameNode is telling me it cannot connect when trying >>> to do >>> a checkpoint: >>>=20 >>> 2008-12-15 09:59:48,017 ERROR >>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in >>> doCheckpoint: >>> 2008-12-15 09:59:48,018 ERROR >>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: >>> java.net.ConnectException: Connection refused >>> at java.net.PlainSocketImpl.socketConnect(Native Method) >>>=20 >>> I changed my masters file to just contain the hostname of the >>> secondarynamenode, this seems to have properly started the NameNode >>> where I >>> launched the ./bin/start-dfs.sh from and started SecondaryNameNode on >>> correct node as well. But it seems to be unable to connect back to >>> primary. >>> I have hadoop-site.xml pointing to fs.default.name of primary, but >>> otherwise >>> there are not links back. Where would I specify to the secondary where >>> primary is located? >>>=20 >>> We're also upgrading to Hadoop 0.19.0 at this time. >>>=20 >>> Thank you for any help. >>>=20 >>> Jonathan Gray >=20 --B_3314272641_5350170--