hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stuart Smith <stu24m...@yahoo.com>
Subject Diagnosing can't find mapper errors
Date Tue, 06 Jul 2010 21:16:36 GMT
Hello,

  It seems like everytime I'm writing a map class, I make some random mistake that triggers
a generically unhelpful ClassNotFoundException that results in a large amount of frustration
from forgetting the static keyword or something silly.

Are there any lint tools to check your class beforehand? Is there a way to make hadoop give
useful error messages when something goes wrong? To catch things at compile time?

My current trouble is with a class like so:

package hbaseScanner;

...

public class Validator 
{
...
	public static class ValidateMapper extends TableMapper<Text,IntWritable>
	{				
		@Override
		public void map( ImmutableBytesWritable key, Result value, Context context )
		throws IOException, InterruptedException
		{
...
	public static class Reduce extends TableReducer<Text,IntWritable,Text>
	{
		@Override
		public void reduce( Text key, Iterable<IntWritable> Values, Context context )
		throws IOException, InterruptedException
		{

...
	public static void main(String[] args) throws Exception 
	{
		Configuration configuration = new Configuration();
		Job job = new Job(configuration);
		
		//don't add anything, and it will scan everything (according to docs)
		Scan scan = new Scan();
		scan.addColumn( Bytes.toBytes("content"), Bytes.toBytes("file") );
		
		TableMapReduceUtil.initTableMapperJob("filestore", scan, ValidateMapper.class, Text.class,
IntWritable.class, job);
		TableMapReduceUtil.initTableReducerJob("statistics", Reduce.class, job);
		
		job.waitForCompletion(true);
	}

}

Hadoop will only say:
us : FAILED
10/07/06 14:09:54 INFO mapred.JobClient: Task Id : attempt_201006211228_0010_m_000010_0, Status
: FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: hbaseScanner.Validator$Map
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
	at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)

I'm running like so:

~/hadoop/bin$ ./hadoop jar ./ScannerValidator.jar hbaseScanner.Validator

*sigh*. So frustrating.


Take care,
  -stu



      

Mime
View raw message