hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stu24m...@yahoo.com
Subject Re: Diagnosing can't find mapper errors
Date Wed, 07 Jul 2010 03:27:28 GMT
Oh no. I just realized this was all posted to hdfs, not hbase - I guess I really needed to
sleep. Sorry everyone! Long weekend.

Best,
 -stu

Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: stu24mail@yahoo.com
Date: Wed, 7 Jul 2010 03:23:52 
To: <hdfs-user@hadoop.apache.org>
Reply-To: hdfs-user@hadoop.apache.org
Subject: Re: Diagnosing can't find mapper errors

Just to follow up: I've confirmed changing the eclipse jar export options results in map tasks
that can run to completion.

Take care,
 -stu

Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: Stuart Smith <stu24mail@yahoo.com>
Date: Tue, 6 Jul 2010 17:12:52 
To: <hdfs-user@hadoop.apache.org>
Reply-To: hdfs-user@hadoop.apache.org
Subject: Re: Diagnosing can't find mapper errors


Ok, I think I figured out this specific issue (while I still don't know how to improve the
experience of "can't find mapper" in general).

On Eclipse, when I said "export to jar", I checked:

"Extract required libraries into generated jar"

instead of:

"Package required libraries into generated jar"

I really have no idea what the difference is, TBH. The distinction between Extracting and
Packaging seems rather cryptic (I did try this: http://www.google.com/search?client=ubuntu&channel=fs&q=eclipse+extracting+packaging+jar&ie=utf-8&oe=utf-8,
but it was unhelpful).

However, I say "think" because now my map/reduce is just stalled: it spun up 256 map jobs
on my 4 node cluster, which just brought everything to a crawl. This seems odd, since most
complaints seem to be in the opposite direction: not enough tasks due to the map task per
region limitation (from what I can tell).

I'll keep digging around, but any advice on optimizing M/R tasks over an hbase table is welcome..

Take care,
  -stu

--- On Tue, 7/6/10, Stuart Smith <stu24mail@yahoo.com> wrote:

> From: Stuart Smith <stu24mail@yahoo.com>
> Subject: Diagnosing can't find mapper errors
> To: hdfs-user@hadoop.apache.org
> Date: Tuesday, July 6, 2010, 5:16 PM
> Hello,
> 
>   It seems like everytime I'm writing a map class, I
> make some random mistake that triggers a generically
> unhelpful ClassNotFoundException that results in a large
> amount of frustration from forgetting the static keyword or
> something silly.
> 
> Are there any lint tools to check your class beforehand? Is
> there a way to make hadoop give useful error messages when
> something goes wrong? To catch things at compile time?
> 
> My current trouble is with a class like so:
> 
> package hbaseScanner;
> 
> ...
> 
> public class Validator 
> {
> ...
>     public static class ValidateMapper
> extends TableMapper<Text,IntWritable>
>     {       
>         
>         @Override
>         public void map(
> ImmutableBytesWritable key, Result value, Context context )
>         throws IOException,
> InterruptedException
>         {
> ...
>     public static class Reduce extends
> TableReducer<Text,IntWritable,Text>
>     {
>         @Override
>         public void reduce(
> Text key, Iterable<IntWritable> Values, Context
> context )
>         throws IOException,
> InterruptedException
>         {
> 
> ...
>     public static void main(String[] args)
> throws Exception 
>     {
>         Configuration
> configuration = new Configuration();
>         Job job = new
> Job(configuration);
>         
>         //don't add anything,
> and it will scan everything (according to docs)
>         Scan scan = new
> Scan();
>         scan.addColumn(
> Bytes.toBytes("content"), Bytes.toBytes("file") );
>         
>        
> TableMapReduceUtil.initTableMapperJob("filestore", scan,
> ValidateMapper.class, Text.class, IntWritable.class, job);
>        
> TableMapReduceUtil.initTableReducerJob("statistics",
> Reduce.class, job);
>         
>        
> job.waitForCompletion(true);
>     }
> 
> }
> 
> Hadoop will only say:
> us : FAILED
> 10/07/06 14:09:54 INFO mapred.JobClient: Task Id :
> attempt_201006211228_0010_m_000010_0, Status : FAILED
> java.lang.RuntimeException:
> java.lang.ClassNotFoundException:
> hbaseScanner.Validator$Map
>     at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>     at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
>     at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
> 
> I'm running like so:
> 
> ~/hadoop/bin$ ./hadoop jar ./ScannerValidator.jar
> hbaseScanner.Validator
> 
> *sigh*. So frustrating.
> 
> 
> Take care,
>   -stu
> 
> 
> 
>       
> 



Mime
View raw message