Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 96302 invoked from network); 4 Sep 2008 13:34:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 4 Sep 2008 13:34:13 -0000 Received: (qmail 13067 invoked by uid 500); 4 Sep 2008 13:34:05 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 13035 invoked by uid 500); 4 Sep 2008 13:34:05 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 13024 invoked by uid 99); 4 Sep 2008 13:34:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Sep 2008 06:34:05 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [203.99.254.143] (HELO rsmtp1.corp.hki.yahoo.com) (203.99.254.143) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Sep 2008 13:33:05 +0000 Received: from [192.168.1.35] (vpn-client-88-77.eglbp.corp.yahoo.com [10.66.88.77]) by rsmtp1.corp.hki.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id m84DXSAr086145 for ; Thu, 4 Sep 2008 06:33:30 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=user-agent:date:subject:from:to:message-id:thread-topic: thread-index:in-reply-to:mime-version:content-type:content-transfer-encoding; b=ottebyw5dQ9voUwJ2/GQjLfw2kQaL0joTiGRYy/9E4buZfdnVa8oupclTj/t5q/D User-Agent: Microsoft-Entourage/12.12.0.080729 Date: Thu, 04 Sep 2008 19:03:27 +0530 Subject: Re: Could not obtain block: blk_-2634319951074439134_1129 file=/user/root/crawl_debug/segments/20080825053518/content/part-00002/data From: Devaraj Das To: Message-ID: Thread-Topic: Could not obtain block: blk_-2634319951074439134_1129 file=/user/root/crawl_debug/segments/20080825053518/content/part-00002/data Thread-Index: AckOks+8BB3IBv+9lUWh0IeHVq/8nw== In-Reply-To: <200809041506.36359.espen@trank.no> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org > I started a profile of the reduce-task. I've attached the profiling output. > It seems from the samples that ramManager.waitForDataToMerge() doesn't > actually wait. > Has anybody seen this behavior. This has been fixed in HADOOP-3940 On 9/4/08 6:36 PM, "Espen Amble Kolstad" wrote: > I have the same problem on our cluster. > > It seems the reducer-tasks are using all cpu, long before there's anything to > shuffle. > > I started a profile of the reduce-task. I've attached the profiling output. > It seems from the samples that ramManager.waitForDataToMerge() doesn't > actually wait. > Has anybody seen this behavior. > > Espen > > On Thursday 28 August 2008 06:11:42 wangxu wrote: >> Hi,all >> I am using hadoop-0.18.0-core.jar and nutch-2008-08-18_04-01-55.jar, >> and running hadoop on one namenode and 4 slaves. >> attached is my hadoop-site.xml, and I didn't change the file >> hadoop-default.xml >> >> when data in segments are large,this kind of errors occure: >> >> java.io.IOException: Could not obtain block: blk_-2634319951074439134_1129 >> file=/user/root/crawl_debug/segments/20080825053518/content/part-00002/data >> at >> org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.jav >> a:1462) at >> org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1 >> 312) at >> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417) at >> java.io.DataInputStream.readFully(DataInputStream.java:178) >> at >> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:64 >> ) at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:102) >> at >> org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1646) >> at >> org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.ja >> va:1712) at >> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java: >> 1787) at >> org.apache.hadoop.mapred.SequenceFileRecordReader.getCurrentValue(SequenceF >> ileRecordReader.java:104) at >> org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordRe >> ader.java:79) at >> org.apache.hadoop.mapred.join.WrappedRecordReader.next(WrappedRecordReader. >> java:112) at >> org.apache.hadoop.mapred.join.WrappedRecordReader.accept(WrappedRecordReade >> r.java:130) at >> org.apache.hadoop.mapred.join.CompositeRecordReader.fillJoinCollector(Compo >> siteRecordReader.java:398) at >> org.apache.hadoop.mapred.join.JoinRecordReader.next(JoinRecordReader.java:5 >> 6) at >> org.apache.hadoop.mapred.join.JoinRecordReader.next(JoinRecordReader.java:3 >> 3) at >> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:165) >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) >> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209) >> >> >> how can I correct this? >> thanks. >> Xu >