hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sridhar Raman" <sridhar.ra...@gmail.com>
Subject Re: Inconsistency while running in eclipse and cygwin
Date Thu, 01 May 2008 12:49:05 GMT
I managed to fix both the problems.

1)  The one in Eclipse was happening because of the df task which wasn't
possible in Windows.  Once I added cygwin/bin to the Path, this started
working.

2)  This one occurred because the output value class of my Combiner and
Reducer was different.

On Wed, Apr 30, 2008 at 6:49 PM, Sridhar Raman <sridhar.raman@gmail.com>
wrote:

> I am trying to run my MR task in a local machine.  I am doing this because
> this is my first foray into MapReduce, and want to debug completely before I
> move it to a multi-node cluser.  The initial hitch that I faced was in
> running it from Eclipse.  Thanks to this response<http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01368.html>,
> I managed to get that working.
>
> But now I am facing a strange problem.
>
> My Mapper takes <IntWritable, Vector> as input, and returns <IntWritable,
> Vector>.
> I have a Combiner that takes the <IntWritable, Iterator<Vector>> and
> computes the intermediate output of <IntWritable, InterVector>.  This
> IntVector basically stores the sum of the Vectors present in the iterator,
> along with other information.
>
> Now to the problem.  If I run this in Eclipse, after the Mapper is done, I
> straightaway get this exception:
> Iteration 0
> java.io.IOException: Job failed!
>     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:831)
>     at
> com.company.analytics.clustering.mr.core.KMeansDriver.runIteration(KMeansDriver.java:144)
>     at
> com.company.analytics.clustering.mr.core.KMeansDriver.runJob(KMeansDriver.java:91)
>     at
> com.company.analytics.clustering.mr.core.KMeansDriver.main(KMeansDriver.java:37)
>
> Whereas if I run this from the cygwin command prompt, the execution
> manages to go to the Combiner, but after that it gives this exception:
> java.io.IOException: wrong value class:
> com.company.analytics.clustering.feature.
> InterVector is not class com.company.analytics.clustering.feature.Vector
>         at
> org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:938)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$1.collect(MapTask.java:414)
>         at
> com.company.analytics.clustering.mr.core.KMeansCombiner.reduce(KMeansCombiner.java:57)
>         at
> com.company.analytics.clustering.mr.core.KMeansCombiner.reduce(KMeansCombiner.java:1)
>
> Any idea what could be the problem.  Both the cygwin and Eclipse take the
> same input folder.
> I would be glad if I get any assistance.
>
> Thanks,
> Sridhar
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message