Return-Path: Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: (qmail 47924 invoked from network); 8 Oct 2009 15:05:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 8 Oct 2009 15:05:14 -0000 Received: (qmail 38784 invoked by uid 500); 8 Oct 2009 15:05:14 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 38729 invoked by uid 500); 8 Oct 2009 15:05:14 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 38720 invoked by uid 99); 8 Oct 2009 15:05:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Oct 2009 15:05:13 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of geoffry.roberts@gmail.com designates 209.85.216.197 as permitted sender) Received: from [209.85.216.197] (HELO mail-px0-f197.google.com) (209.85.216.197) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Oct 2009 15:05:02 +0000 Received: by pxi35 with SMTP id 35so6126394pxi.2 for ; Thu, 08 Oct 2009 08:04:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=3mc1q/QDjefGr+MdN/0ZsSPPw5kEzDCkvYLBaw2bFMY=; b=dnJpfjifWcK0HwN8L7SomQIs82oc0cNhCxqwbNkVfvflzg43AbiKge9EBnLO/4MjV1 wj7u6rNDR4yZnk8F5EeVlKS4LAC9sLo4UTRwf06ByfjX20a9urLdIlnXp7v8L+VxPyMc xGalurmS7/f9EPWGHMKrncW/ZPtx6YBYyjG+o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=c56Dg9q5X/fAt78G0eXPa9EWWwN74Wo9dK+KnSiYvSoLlx1ALBi93udtedS6U+BBRr KwX9emtHef1GVHOP9rCZYX/enJArhckGswEzjElStVGXYwOTpdpIuLyPh/AoNieB+9LZ nSiPTsudycOmqFMXKwlfY3LsM1/hGqZti0vxo= MIME-Version: 1.0 Received: by 10.142.9.29 with SMTP id 29mr106663wfi.131.1255014281436; Thu, 08 Oct 2009 08:04:41 -0700 (PDT) In-Reply-To: <314098690910080701l6c986949ke75583bba0557932@mail.gmail.com> References: <9ec65d4c0910071352x39c20f0di9a7f4f4b6a964cea@mail.gmail.com> <314098690910080701l6c986949ke75583bba0557932@mail.gmail.com> Date: Thu, 8 Oct 2009 07:04:41 -0800 Message-ID: <9ec65d4c0910080804y6f7c1681xee1db728a67b33cf@mail.gmail.com> Subject: Re: MapRed Job Completes; Output Ceases Mid-Job From: Geoffry Roberts To: mapreduce-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=00504502b8516edd0804756dc891 X-Virus-Checked: Checked by ClamAV on apache.org --00504502b8516edd0804756dc891 Content-Type: text/plain; charset=ISO-8859-1 Jason, Quite possibly, here's what I did: I upped "dfs.datanode.max.xcievers" to 512, which is a doubling, and the full set of output files are created correctly. Thanks for responding. Learning, learning the ins and outs of Hadoop. On Thu, Oct 8, 2009 at 6:01 AM, Jason Venner wrote: > Are you perhaps creating large numbers of files, and running out of file > descriptors in your tasks. > > > On Wed, Oct 7, 2009 at 1:52 PM, Geoffry Roberts > wrote: > >> All, >> >> I have a MapRed job that ceases to produce output about halfway through. >> The obvious question is why? >> >> This job reads a file and uses MultipleTextOutputFormat to generate output >> files named with the output key. At about the halfway point, the job >> continues to create files, but they are all of zero length. I've worked >> with this input file extensively and I know it actually contains the >> required data and that it is clean or at least it was when I copied it in. >> >> My first impulse was to check for a full disk, but there seems to be ample >> free space. >> >> This doesn't appear to have anything to do with my code. >> >> stderror is full of the following entry: >> >> java.io.EOFException >> >> >> at java.io.DataInputStream.readByte(DataInputStream.java:250) >> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298) >> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319) >> at org.apache.hadoop.io.Text.readString(Text.java:400) >> >> >> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2837) >> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2762) >> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2046) >> >> >> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2232) >> >> >> syslog for the reducer starts filling up with the following at what could >> indeed be the halfway point: >> >> 2009-10-07 11:27:50,874 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException >> >> >> 2009-10-07 11:27:50,916 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-1693260904457793456_3495 >> 2009-10-07 11:27:56,919 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException >> >> >> 2009-10-07 11:27:56,919 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_7536254999085848659_3495 >> 2009-10-07 11:28:02,921 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException >> >> >> 2009-10-07 11:28:02,921 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-7513223558440754487_3495 >> 2009-10-07 11:28:08,924 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException >> >> >> 2009-10-07 11:28:08,924 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_2580888829875117043_3495 >> 2009-10-07 11:28:14,965 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block. >> >> >> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2781) >> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2046) >> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2232) >> >> >> > > > -- > Pro Hadoop, a book to guide you from beginner to hadoop mastery, > http://www.amazon.com/dp/1430219424?tag=jewlerymall > www.prohadoopbook.com a community for Hadoop Professionals > --00504502b8516edd0804756dc891 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Jason,

Quite possibly, here's what I did: I upped "dfs.data= node.max.xcievers" to 512, which is a doubling, and the full set of ou= tput files are created correctly.=A0

Thanks for responding.

Learning, learning the ins and outs of Hadoop.

On Thu, Oct 8, 2009 at 6:01 AM, Jason Venner <jason.hadoop@gmail.com> wrote:
Are you perhaps c= reating large numbers of files, and running out of file descriptors in your= tasks.


On Wed, Oct= 7, 2009 at 1:52 PM, Geoffry Roberts <geoffry.roberts@gmail.com> wrote:
All,

I hav= e a MapRed job that ceases to produce output about halfway through.=A0 The = obvious question is why?

This job reads a file and uses MultipleTextOutputFormat to generate out= put files named with the output key.=A0 At about the halfway point, the job= continues to create files, but they are all of zero length.=A0=A0=A0 I'= ;ve worked with this input file extensively and I know it actually contains= the required data and that it is clean or at least it was when I copied it= in.

My first impulse was to check for a full disk, but there seems to be am= ple free space.

This doesn't appear to have anything to do with = my code.

stderror is full of the following entry:
java.io.EO=
FException



	at java.io.DataInputStream.readByte(DataInputStream.java:250)
	at org.a=
pache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
	at org.=
apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
	at org.=
apache.hadoop.io.Text.readString(Text.java:400)



	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStrea=
m(DFSClient.java:2837)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStr=
eam.nextBlockOutputStream(DFSClient.java:2762)
	at org.apache.hadoop.hdf=
s.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2046)



	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCl=
ient.java:2232)

syslog for the reducer starts filling up with = the following at what could indeed be the halfway point:
2009-10-07=
 11:27:50,874 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlo=
ckOutputStream java.io.EOFException



2009-10-07 11:27:50,916 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning b=
lock blk_-1693260904457793456_3495
2009-10-07 11:27:56,919 INFO org.apac=
he.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.EOFE=
xception



2009-10-07 11:27:56,919 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning b=
lock blk_7536254999085848659_3495
2009-10-07 11:28:02,921 INFO org.apach=
e.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.EOFEx=
ception



2009-10-07 11:28:02,921 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning b=
lock blk_-7513223558440754487_3495
2009-10-07 11:28:08,924 INFO org.apac=
he.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.EOFE=
xception



2009-10-07 11:28:08,924 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning b=
lock blk_2580888829875117043_3495
2009-10-07 11:28:14,965 WARN org.apach=
e.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unabl=
e to create new block.



	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(=
DFSClient.java:2781)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStrea=
m.access$2000(DFSClient.java:2046)
	at org.apache.hadoop.hdfs.DFSClient$=
DFSOutputStream$DataStreamer.run(DFSClient.java:2232)




--
Pro Hadoop, a book to guide you from beginner to hadoop master= y,
http://www.amazon.com/dp/1430219424?tag=3Djewlerymall www.prohadoopboo= k.com a community for Hadoop Professionals

--00504502b8516edd0804756dc891--