hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: Solr + Hadoop
Date Thu, 13 Jan 2011 13:01:53 GMT
The second issue is cause your Mapper code is unable to find the solr
and other user classes (It does not have a JAR on its classpath to
load one from). You can avoid this by making sure that (either):

1. Your submission code is doing a Job.setJarByClass(...) or
Job.setJar(...) --> This will send the JAR via the DistributedCache to
all the TaskTrackers that will run the Mappers. You can also manually
add required files to DistributedCache via it's API.

2. Add your Solr + other library jars to Hadoop's lib/
directory/classpath across all machines and restart the cluster.

The latter is the way to go if you will always need some classes in your setup.

On Thu, Jan 13, 2011 at 5:48 PM, Joan <joan.monplet@gmail.com> wrote:
> Hi,
>
> I'm trying build solr index with MapReduce (Hadoop) and I'm using
> https://issues.apache.org/jira/browse/SOLR-1301 but I've a problem with
> hadoop version and this patch.
>
> When I compile this patch, I use 0.21.0 hadoop version, I don't have any
> problem but when I'm trying to run my job in Hadoop (0.0.21) I get some
> error like this:
>
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>  at
> org.apache.solr.hadoop.SolrOutputFormat.checkOutputSpecs(SolrOutputFormat.java:147)
>         at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:373)
>         at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:334)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:960)
>         at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:976)
>
>
>
> I try to override the next method:
>
>   public void checkOutputSpecs(JobContext job) throws IOException {
>     super.checkOutputSpecs(job);
>     if (job.getConfiguration().get(SETUP_OK) == null) {
>       throw new IOException("Solr home cache not set up!");
>     }
>   }
>
> by
>
>   public void checkOutputSpecs(Job job) throws IOException {
>     super.checkOutputSpecs(job);
>     if (job.getConfiguration().get(SETUP_OK) == null) {
>       throw new IOException("Solr home cache not set up!");
>     }
>   }
>
> but I continue receive some error:
>
>         java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.solr.hadoop.SolrOutputFormat
>         at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1128)
>         at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:203)
>         at org.apache.hadoop.mapred.Task.initialize(Task.java:487)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>         at java.security.AccessController.doPrivileged(Native Method)
>
>
>
> Please, Is someone using this patch with 0.21.0 Version Hadoop?. Can someone
> help me?
>
> Thanks,
>
> Joan
>



-- 
Harsh J
www.harshj.com

Mime
View raw message