Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 53788 invoked from network); 6 Dec 2008 02:50:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 6 Dec 2008 02:50:42 -0000 Received: (qmail 94722 invoked by uid 500); 6 Dec 2008 02:50:49 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 94672 invoked by uid 500); 6 Dec 2008 02:50:49 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 94659 invoked by uid 99); 6 Dec 2008 02:50:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Dec 2008 18:50:49 -0800 X-ASF-Spam-Status: No, hits=-4.0 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [129.93.181.2] (HELO mathstat.unl.edu) (129.93.181.2) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Dec 2008 02:49:19 +0000 Received: from [192.168.0.103] (user-0cdvqpp.cable.mindspring.com [24.223.235.57]) (authenticated bits=0) by mathstat.unl.edu (8.13.8/8.13.8) with ESMTP id mB62ktfu004112 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) for ; Fri, 5 Dec 2008 20:46:58 -0600 Message-Id: From: Brian Bockelman To: core-user@hadoop.apache.org In-Reply-To: <784136.92435.qm@web56201.mail.re3.yahoo.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Subject: Re: Block not found during commitBlockSynchronization Date: Fri, 5 Dec 2008 20:46:55 -0600 References: <8980254B-D22A-4881-AE31-2A8FE5E813F5@math.unl.edu> <784136.92435.qm@web56201.mail.re3.yahoo.com> X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org This is 0.19.0. Grepping around, it appears that message for this block has been printed 1-5 Hz throughout all our logs (oldest logs are 12-3). Has happened about .5 million times. If I grep for the "nextGenerationStamp" error message, it's happened .4M times. Anything else I can provide? Brian On Dec 5, 2008, at 8:31 PM, Tsz Wo (Nicholas), Sze wrote: > Which version are you using? > > Calling commitBlockSynchronization(...) with newgenerationstamp=0, > newlength=0, newtargets=[] does not look normal. You may check the > namenode log and the client log about the block > blk_-4236881263392665762. > > Nicholas Sze > > > > > ----- Original Message ---- >> From: Brian Bockelman >> To: core-user@hadoop.apache.org >> Sent: Friday, December 5, 2008 5:22:03 PM >> Subject: Block not found during commitBlockSynchronization >> >> Hey, >> >> I'm seeing this message repeated over and over in my logs: >> >> 2008-12-05 19:20:00,534 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: >> commitBlockSynchronization(lastblock=blk_-4236881263392665762_88597, >> newgenerationstamp=0, newlength=0, newtargets=[]) >> 2008-12-05 19:20:00,534 INFO org.apache.hadoop.ipc.Server: IPC >> Server handler 29 >> on 9000, call >> commitBlockSynchronization(blk_-4236881263392665762_88597, 0, 0, >> false, true, >> [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@67537412) from >> 172.16.1.184:57586: error: java.io.IOException: Block >> (=blk_-4236881263392665762_88597) not found >> java.io.IOException: Block (=blk_-4236881263392665762_88597) not >> found >> at >> org >> .apache >> .hadoop >> .hdfs >> .server >> .namenode.FSNamesystem.commitBlockSynchronization(FSNamesystem.java: >> 1898) >> at >> org >> .apache >> .hadoop >> .hdfs >> .server.namenode.NameNode.commitBlockSynchronization(NameNode.java: >> 410) >> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) >> at >> sun >> .reflect >> .DelegatingMethodAccessorImpl >> .invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892) >> >> What can I do to debug? >> >> Brian