hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chenlbspace" <chenlbsp...@sina.com>
Subject Re: Re: hadoop index org/apache/lucene/store/Directory NoClassDefFoundError
Date Fri, 10 Oct 2008 09:50:04 GMT
hi, imcaptor 

I try to cp lucene-core-2.3.1.jar to lib, but error also at map task.

-------------------lib-----------------
[chenlb@master hadoop]$ pwd
/home/chenlb/hadoop
[chenlb@master hadoop]$ ls lib
commons-cli-2.0-SNAPSHOT.jar  commons-logging-1.0.4.jar      jetty-5.1.4.jar          junit-3.8.1.jar
         kfs-0.1.LICENSE.txt    native
commons-codec-1.3.jar         commons-logging-api-1.0.4.jar  jetty-5.1.4.LICENSE.txt  junit-3.8.1.LICENSE.txt
 log4j-1.2.13.jar       servlet-api.jar
commons-httpclient-3.0.1.jar  jets3t-0.5.0.jar               jetty-ext                kfs-0.1.jar
             lucene-core-2.3.1.jar  xmlenc-0.52.jar

-------------------hadoop-site.xml---------------------------------
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
        <name>fs.default.name</name>
        <value>hdfs://master:9000/</value>
</property>
<property>
        <name>mapred.job.tracker</name>
        <value>hdfs://master:9001/</value>
</property>
<property>
        <name>dfs.replication</name>
        <value>1</value>
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>/home/chenlb/hadoop/tmp/</value>
</property>
</configuration>
----------------------------

Thinks!

=====================error=======================
[chenlb@master hadoop]$ bin/hadoop jar hadoop-0.17.1-index.jar -inputPaths data-in -outputPath
index-msg-out -indexPath index -conf index-config.xml
08/10/09 22:33:36 INFO main.UpdateIndex: inputPaths = data-in
08/10/09 22:33:36 INFO main.UpdateIndex: outputPath = index-msg-out
08/10/09 22:33:36 INFO main.UpdateIndex: shards     = null
08/10/09 22:33:36 INFO main.UpdateIndex: indexPath  = index
08/10/09 22:33:36 INFO main.UpdateIndex: numShards  = -1
08/10/09 22:33:36 INFO main.UpdateIndex: numMapTasks= -1
08/10/09 22:33:36 INFO main.UpdateIndex: confPath   = index-config.xml
08/10/09 22:33:37 INFO main.UpdateIndex: sea.index.updater = org.apache.hadoop.contrib.index.mapred.IndexUpdater
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.input.dir = hdfs://master:9000/user/chenlb/data-in
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.output.dir = hdfs://master:9000/user/chenlb/index-msg-out
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.map.tasks = 2
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.reduce.tasks = 1
08/10/09 22:33:37 INFO mapred.IndexUpdater: 1 shards = -1@index/00000@-1
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.input.format.class = org.apache.hadoop.contrib.index.example.LineDocInputFormat
08/10/09 22:33:38 INFO mapred.FileInputFormat: Total input paths to process : 2
08/10/09 22:33:39 INFO mapred.JobClient: Running job: job_200810092201_0010
08/10/09 22:33:40 INFO mapred.JobClient:  map 0% reduce 0%
08/10/09 22:33:48 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000000_0, Status
: FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
        at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        ... 9 more
08/10/09 22:33:50 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000001_0, Status
: FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
        at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        ... 9 more
08/10/09 22:33:55 INFO mapred.JobClient: Task Id : task_200810092201_0010_r_000000_0, Status
: FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
        at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:964)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:327)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        ... 9 more
08/10/09 22:33:56 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000000_1, Status
: FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
        at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        ... 9 more
08/10/09 22:33:59 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000001_1, Status
: FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
        at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        ... 9 more
08/10/09 22:34:03 INFO mapred.JobClient: Task Id : task_200810092201_0010_r_000000_1, Status
: FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
        at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:964)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:327)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        ... 9 more
08/10/09 22:34:05 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000000_2, Status
: FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
        at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        ... 9 more
08/10/09 22:34:08 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000001_2, Status
: FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
        at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        ... 9 more
08/10/09 22:34:10 INFO mapred.JobClient: Task Id : task_200810092201_0010_r_000000_2, Status
: FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
        at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:964)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:327)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        ... 9 more
08/10/09 22:34:13 INFO mapred.JobClient:  map 100% reduce 100%
08/10/09 22:34:14 INFO main.UpdateIndex: Elapsed time is  37s
Elapsed time is 37s
java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062)
        at org.apache.hadoop.contrib.index.mapred.IndexUpdater.run(IndexUpdater.java:53)
        at org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:263)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)

2008-10-10 



chenlbspace 



发件人: imcaptor 
发送时间: 2008-10-10  17:07:39 
收件人: core-user@hadoop.apache.org 
抄送: 
主题: Re: hadoop index org/apache/lucene/store/Directory NoClassDefFoundError 
 
Try do this.
In the hadoop path,
-bash-3.00$ pwd
/data/hadoop/hadoop-0.18.1
cp src/contrib/index/lib/lucene-core-2.3.1.jar lib
Then you can run the task.
chenlbspace 写道:
> hi, 
>
> I try to use contrib/index/hadoop-0.17.1-index.jar build lucene index, but Directory
NoClassDefFoundError.
>
> How solve? Thinks.
>
> Additional, hadoop run Pseudo-Distributed Mode. and can run "bin/hadoop jar hadoop-*-examples.jar
grep input output 'dfs[a-z.]+'"
>
> ---------------------------error----------------------------------------------------------------------------------
> [chenlb@master hadoop]$ bin/hadoop jar contrib/index/hadoop-0.17.1-index.jar -inputPaths
src/contrib/index/sample -outputPath index-msg-out -indexPath index -conf src/contrib/index/conf/index-config.xml
> 08/10/09 21:41:28 INFO main.UpdateIndex: inputPaths = src/contrib/index/sample
> 08/10/09 21:41:28 INFO main.UpdateIndex: outputPath = index-msg-out
> 08/10/09 21:41:28 INFO main.UpdateIndex: shards     = null
> 08/10/09 21:41:28 INFO main.UpdateIndex: indexPath  = index
> 08/10/09 21:41:28 INFO main.UpdateIndex: numShards  = -1
> 08/10/09 21:41:28 INFO main.UpdateIndex: numMapTasks= -1
> 08/10/09 21:41:28 INFO main.UpdateIndex: confPath   = src/contrib/index/conf/index-config.xml
> 08/10/09 21:41:28 INFO main.UpdateIndex: Elapsed time is  0s
> Elapsed time is 0s
> java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:247)
>         at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
>         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
>         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
>         at org.apache.hadoop.contrib.index.mapred.IndexUpdateConfiguration.getIndexUpdaterClass(IndexUpdateConfiguration.java:145)
>         at org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:257)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
>         at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
> Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>         at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>         ... 16 more
>
> -----------------------------------------------------------------------------------------------------------------
>
> 2008-10-10 
>
>
>
> chenlbspace 
>
>   
__________ NOD32 3510 (20081010) Information __________
This message was checked by NOD32 antivirus system.
http://www.nod32cn.com
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message