hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sridhar Raman" <sridhar.ra...@gmail.com>
Subject Inconsistency while running in eclipse and cygwin
Date Wed, 30 Apr 2008 13:19:17 GMT
I am trying to run my MR task in a local machine.  I am doing this because
this is my first foray into MapReduce, and want to debug completely before I
move it to a multi-node cluser.  The initial hitch that I faced was in
running it from Eclipse.  Thanks to this
response<http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01368.html>,
I managed to get that working.

But now I am facing a strange problem.

My Mapper takes <IntWritable, Vector> as input, and returns <IntWritable,
Vector>.
I have a Combiner that takes the <IntWritable, Iterator<Vector>> and
computes the intermediate output of <IntWritable, InterVector>.  This
IntVector basically stores the sum of the Vectors present in the iterator,
along with other information.

Now to the problem.  If I run this in Eclipse, after the Mapper is done, I
straightaway get this exception:
Iteration 0
java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:831)
    at
com.company.analytics.clustering.mr.core.KMeansDriver.runIteration(KMeansDriver.java:144)
    at
com.company.analytics.clustering.mr.core.KMeansDriver.runJob(KMeansDriver.java:91)
    at
com.company.analytics.clustering.mr.core.KMeansDriver.main(KMeansDriver.java:37)

Whereas if I run this from the cygwin command prompt, the execution manages
to go to the Combiner, but after that it gives this exception:
java.io.IOException: wrong value class:
com.company.analytics.clustering.feature.
InterVector is not class com.company.analytics.clustering.feature.Vector
        at
org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:938)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$1.collect(MapTask.java:414)
        at
com.company.analytics.clustering.mr.core.KMeansCombiner.reduce(KMeansCombiner.java:57)
        at
com.company.analytics.clustering.mr.core.KMeansCombiner.reduce(KMeansCombiner.java:1)

Any idea what could be the problem.  Both the cygwin and Eclipse take the
same input folder.
I would be glad if I get any assistance.

Thanks,
Sridhar

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message