Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of standingbear@gmail.com
 designates 72.14.220.154 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=D8+UonP4aZaa60UzBwYmUhwnEHYv7o8Nreb/m+g0y+TW/6a2KABgCNUfAg7VmN0pRa1S1ePXlzlwn3kIdAaN5dsEXC9SmstmD4wuoi12EmLd+NsfIMDDrujIbG1oIAkSHGSXaIreOoGctopD8Lfy/RUUEgafRYHciVl/LxHwyc8=
Message-ID: <bb7e77f10801160536x10865a36sff6a032d609d7890@mail.gmail.com>
Date: Wed, 16 Jan 2008 08:36:09 -0500
From: "Jim the Standing Bear" <standingbear@gmail.com>
To: hadoop-user@lucene.apache.org
Subject: Re: unable to figure out this exception from reduce task
In-Reply-To: 
 <60499C890DBB8042BC7834CC82FB23798C0FAE@SNV-EXVS09.ds.corp.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <bb7e77f10801152153u293b0bb1y44a2028400e32d44@mail.gmail.com>
	 <F4CB27F4-6A0C-4099-A619-D91DB51FBFD4@gmail.com>
	 <60499C890DBB8042BC7834CC82FB23798C0FAE@SNV-EXVS09.ds.corp.yahoo.com>

Thanks Runping.  It seems the bug is still open.  However, in my case,
there were more than plenty of disk space available.


On Jan 16, 2008 2:44 AM, Runping Qi <runping@yahoo-inc.com> wrote:
>
> I encountered a similar case.
> Here is the Jira: https://issues.apache.org/jira/browse/HADOOP-2164
>
> Runping
>
>
>
> > -----Original Message-----
> > From: Vadim Zaliva [mailto:krokodil@gmail.com]
> > Sent: Tuesday, January 15, 2008 9:59 PM
> > To: hadoop-user@lucene.apache.org
> > Subject: Re: unable to figure out this exception from reduce task
> >
> > On Jan 15, 2008, at 21:53, Jim the Standing Bear wrote:
> >
> > I was asking lot of questions today, so I am glad to contribute at
> > least one answer. I have this problem when there was type mismatch
> > for key or values. You need to set up right type at your JobConf like
> > this:
> >
> >          conf.setOutputKeyClass(Text.class);
> >          conf.setOutputValueClass(LongWritable.class);
> >
> > (using appropriate types our mapper produce)
> >
> > Vadim
> >
> > > I am using hadoop 0.15.1 to index some catalog that has a tree-like
> > > structure, where the leaf nodes are data files.  My main task is a
> > > loop that performs a breadth-first walkthrough that parses out URLs
> to
> > > catalogs and datafiles at the next level, which is done in a mapper.
> > > To determine when the loop should terminate, I use a reduce task
> that
> > > counts the number of new catalogs found, and stops the loop when the
> > > count is 0.
> > >
> > > But while I was running the jobs, I kept getting this exception
> > > (pasted below from the logs).  I didn't quite understand what it was
> > > trying to say.  But in my code, I never used LongWritable.  Only
> Text
> > > for output key and output values, and KeyValueTextInputFormat for
> > > input.
> > >
> > > What's weirder is that this exception occurs at different places
> from
> > > job to job.  Sometimes it may be thrown at the 2nd iteration of my
> > > loop, while other times, it may be the 3rd, the 4th etc.  Can
> someone
> > > explain to me what and why this is?  Also, what would be the best
> way
> > > to test/debug a hadoop job??  Thanks.
> > >
> > >
> > > 2008-01-16 00:37:19,941 INFO org.apache.hadoop.mapred.ReduceTask:
> > > task_200801160024_0011_r_000000_1 Copying
> > > task_200801160024_0011_m_000000_0 output from ginkgo.mycluster.org
> > > 2008-01-16 00:37:19,953 INFO org.apache.hadoop.mapred.ReduceTask:
> > > task_200801160024_0011_r_000000_1 done copying
> > > task_200801160024_0011_m_000000_0 output from ginkgo.mycluster.org
> > > 2008-01-16 00:37:19,955 INFO org.apache.hadoop.mapred.ReduceTask:
> > > task_200801160024_0011_r_000000_1 Copying of all map outputs
> complete.
> > > Initiating the last merge on the remaining files in
> > > ramfs://mapoutput26453615
> > > 2008-01-16 00:37:20,088 WARN org.apache.hadoop.mapred.ReduceTask:
> > > task_200801160024_0011_r_000000_1 Final merge of the inmemory files
> > > threw an exception: java.io.IOException: java.io.IOException: wrong
> > > key class: class org.apache.hadoop.io.LongWritable is not class
> > > org.apache.hadoop.io.Text
> > >     at org.apache.hadoop.io.SequenceFile$Sorter
> > > $SegmentDescriptor.nextRawKey(SequenceFile.java:2874)
> > >     at org.apache.hadoop.io.SequenceFile$Sorter
> > > $MergeQueue.merge(SequenceFile.java:2683)
> > >     at
> org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:
> > > 2437)
> > >     at org.apache.hadoop.mapred.ReduceTask
> > > $ReduceCopier.fetchOutputs(ReduceTask.java:1153)
> > >     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:252)
> > >     at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:
> > > 1760)
> > >
> > >     at org.apache.hadoop.mapred.ReduceTask
> > > $ReduceCopier.fetchOutputs(ReduceTask.java:1161)
> > >     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:252)
> > >     at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:
> > > 1760)
> > >
> > > 2008-01-16 00:37:20,090 WARN org.apache.hadoop.mapred.TaskTracker:
> > > Error running child
> > > java.io.IOException: task_200801160024_0011_r_000000_1The reduce
> > > copier failed
> > >     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:253)
> > >     at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:
> > > 1760)
> > >
> > >
> > >
> > > --
> > > --------------------------------------
> > > Standing Bear Has Spoken
> > > --------------------------------------
>
>


-- 
--------------------------------------
Standing Bear Has Spoken
--------------------------------------