hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Kozlov <ale...@cloudera.com>
Subject Re: error:Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzopCodec
Date Wed, 28 Jul 2010 23:34:45 GMT
Hi Alex,

There seems to be a problem with your configuration.  Can you try and check
on which node the attempt_201007202234_0001_m_000000_0 was run?  Go to this
machine and check the config files and the lib subdirectory (for the
presence of the correct configuration and the hadoop-lzo-0.4.4.jar).
Restart the TT and using 'ps -aef | grep -i tasktracker' check that the
hadoop-lzo-0.4.4.jar is in the classpath.

I have a strong suspicion you have stray config files:
com.hadoop.compression.lzo.LzopCodec is not mentioned in the ones you
provided...

Alex K

On Wed, Jul 28, 2010 at 7:42 AM, Alex Luya <alexander.luya@gmail.com> wrote:

> Hello:
>    I got source code from http://github.com/kevinweil/hadoop-lzo,compiled
> them successfully,and then
> 1,copy hadoop-lzo-0.4.4.jar to directory:$HADOOP_HOME/lib of each master
> and
> slave
> 2,Copy all files under directory:../Linux-amd64-64/lib to directory:
> $HADDOOP_HOME/lib/native/Linux-amd64-64 of each master and slave
> 3,and upload a file:test.lzo to HDFS
> 4,then run:hadoop jar $HADOOP_HOME/lib/hadoop-lzo-0.4.4.jar
> com.hadoop.compression.lzo.DistributedLzoIndexer test.lzo to test
>
> got errors:
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 10/07/20 22:37:37 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
> 10/07/20 22:37:37 INFO lzo.LzoCodec: Successfully loaded & initialized
> native-
> lzo library [hadoop-lzo rev 5c25e0073d3dae9ace4bd9eba72e4dc43650c646]
> ##########^_^^_^^_^^_^^_^^_^##################
> (I think this says all native library got loaded successfully)
> ################################
> 10/07/20 22:37:37 INFO lzo.DistributedLzoIndexer: Adding LZO file
> target.lz:o
> to indexing list (no index currently exists)
> ...
> attempt_201007202234_0001_m_000000_0, Status : FAILED
> java.lang.IllegalArgumentException: Compression codec
> com.hadoop.compression.lzo.LzopCodec
>                 not found.
>        at
>
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
>        at
>
> org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
>        at
>
> com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:48)
>        at
>
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ClassNotFoundException:
> com.hadoop.compression.lzo.LzopCodec
>
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>        at java.lang.Class.forName0(Native Method)
>        at java.lang.Class.forName(Class.java:247)
>        at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
>        at
>
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89)
>        ... 6 more
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> There is a installation instruction in this
> link:http://github.com/kevinweil/hadoop-lzo,it says other configurings are
> needed :
>
> Once the libs are built and installed, you may want to add them to the
> class
> paths and library paths. That is, in hadoop-env.sh, set
>
>   (1)export HADOOP_CLASSPATH=/path/to/your/hadoop-lzo-lib.jar
>
> Question:I have copied hadoop-lzo-0.4.4.jar to $HADOOP_HOME/lib,
> ,should I do set this entry like this again? actually, after I add this:
> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_HOME/hbase-0.20.4.jar:
> $HABSE_HOME/config:$ZOOKEEPER_HOME/zookeeper-3.3.1.jar:$HADOOP_HOME/lib
> /hadoop-lzo-0.4.4.jar,redo 1-4 as above,same problem as before,so:
>  how can I
> get hadoop to load hadoop-lzo-0.4.4.jar?)
>
>
>    (2),export JAVA_LIBRARY_PATH=/path/to/hadoop-lzo-native-
> libs:/path/to/standard-hadoop-native-libs
>    Note that there seems to be a bug in /path/to/hadoop/bin/hadoop; comment
> out the line
>    (3)JAVA_LIBRARY_PATH=''
>
>
> Question:since native library got loaded successfully,aren't these
> operation(2)(3) needed?
>
>
> -----------------------------------------------
> I am using hadoop 0.20.2
> core-site.xml
>
> -----------------------------------------------------------------------------
> <configuration>
>        <property>
>                <name>fs.default.name</name>
>                <value>hdfs://hadoop:8020</value>
>        </property>
>        <property>
>                <name>hadoop.tmp.dir</name>
>                <value>/home/hadoop/tmp</value>
>        </property>
>
>        <property>
>                <name>io.compression.codecs</name>
>
>
> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec
>                </value>
>        </property>
>        <property>
>                <name>io.compression.codec.lzo.class</name>
>                <value>com.hadoop.compression.lzo.LzoCodec</value>
>        </property>
> </configuration>
>
>
> -----------------------------------------------------------------------------
> mapred-site.xml
>
> -----------------------------------------------------------------------------
> <configuration>
>        <property>
>                <name>mapred.job.tracker</name>
>                <value>AlexLuya:9001</value>
>        </property>
>        <property>
>                <name>mapred.tasktracker.reduce.tasks.maximum</name>
>                <value>1</value>
>        </property>
>        <property>
>                <name>mapred.tasktracker.map.tasks.maximum</name>
>                <value>1</value>
>        </property>
>        <property>
>                <name>mapred.local.dir</name>
>                <value>/home/alex/hadoop/mapred/local</value>
>        </property>
>        <property>
>                <name>mapred.system.dir</name>
>                <value>/tmp/hadoop/mapred/system</value>
>        </property>
>        <property>
>                <name>mapreduce.map.output.compress</name>
>                <value>true</value>
>        </property>
>        <property>
>                <name>mapreduce.map.output.compress.codec</name>
>                <value>com.hadoop.compression.lzo.LzoCodec</value>
>        </property>
> </configuration>
>
>
> -----------------------------------------------------------------------------
> hadoop-env.sh
>
> -----------------------------------------------------------------------------
> # Set Hadoop-specific environment variables here.
>
> # The only required environment variable is JAVA_HOME.  All others are
> # optional.  When running a distributed configuration it is best to
> # set JAVA_HOME in this file, so that it is correctly defined on
> # remote nodes.
>
> # The java implementation to use.  Required.
> export JAVA_HOME=/usr/local/hadoop/jdk1.6.0_20
>
> # Extra Java CLASSPATH elements.  Optional.
> # export HADOOP_CLASSPATH=
>
> # The maximum amount of heap to use, in MB. Default is 1000.
> export HADOOP_HEAPSIZE=200
>
> # Extra Java runtime options.  Empty by default.
> #export HADOOP_OPTS=-server
>
> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_HOME/hbase-0.20.4.jar:
>
> $HABSE_HOME/config:$ZOOKEEPER_HOME/zookeeper-3.3.1.jar:$HADOOP_HOME/lib/hadoop-
> lzo-0.4.4.jar
> #export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native/Linux-amd64-64
>
> # Command specific options appended to HADOOP_OPTS when specified
> export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_NAMENODE_OPTS"
> export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_SECONDARYNAMENODE_OPTS"
> export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_DATANODE_OPTS"
> export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_BALANCER_OPTS"
> export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_JOBTRACKER_OPTS"
> # export HADOOP_TASKTRACKER_OPTS=
> # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
> # export HADOOP_CLIENT_OPTS
>
> # Extra ssh options.  Empty by default.
> # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
>
> # Where log files are stored.  $HADOOP_HOME/logs by default.
> # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
>
> # File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
> # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
>
> # host:path where hadoop code should be rsync'd from.  Unset by default.
> # export HADOOP_MASTER=master:/home/$USER/src/hadoop
>
> # Seconds to sleep between slave commands.  Unset by default.  This
> # can be useful in large clusters, where, e.g., slave rsyncs can
> # otherwise arrive faster than the master can service them.
> # export HADOOP_SLAVE_SLEEP=0.1
>
> # The directory where pid files are stored. /tmp by default.
> # export HADOOP_PID_DIR=/var/hadoop/pids
>
> # A string representing this instance of hadoop. $USER by default.
> #export HADOOP_IDENT_STRING=$USER
>
> # The scheduling priority for daemon processes.  See 'man nice'.
> # export HADOOP_NICENESS=10
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message