hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Kozlov <ale...@cloudera.com>
Subject Re: How to get lzo compression library loaded?
Date Sun, 01 Aug 2010 07:35:55 GMT
Hi Alex,

This does not seem to be a problem with LZO.  Can you check if
hadoop-0.20.2-core.jar is on the classpath when you execute your command
(with 'ps -aef') and that hadoop-0.20.2-core.jar contains
org.apache.hadoop.io.compress.GzipCodec  (with 'jar tvf
hadoop-0.20.2-core.jar').

Thanks,

Alex K

On Sat, Jul 31, 2010 at 11:27 PM, <stu24mail@yahoo.com> wrote:

> Did you install liblzo2?
> I don't see that listed..
>
> Take care,
>  -stu
> -----Original Message-----
> From: Alex Luya <alexander.luya@gmail.com>
> Date: Sun, 1 Aug 2010 14:15:39
> To: <hdfs-user@hadoop.apache.org>
> Reply-To: hdfs-user@hadoop.apache.org
> Subject: Re: How to get lzo compression library loaded?
>
> Hi,
>    what I am trying to run is:
>
> ---------------------------------------------------------------------------------------------------------------------------
> hadoop jar /usr/local/hadoop/hadoop-0.20.2/lib/hadoop-lzo-0.4.4.jar
> com.hadoop.compression.lzo.DistributedLzoIndexer target.lzo
>
> ---------------------------------------------------------------------------------------------------------------------------
>
> env | grep -i hadoop
>
> ---------------------------------------------------------------------------------------------------------------------------
> NUTCH_HOME=/usr/local/hadoop/nutch-1.1
> HADOOP_HOME=/usr/local/hadoop/hadoop-0.20.2
> HBASE_HOME=/usr/local/hadoop/hbase-0.20.4
>
> PATH=/media/Work/workspace/HDScript:/usr/local/hadoop/nutch-1.1/bin:/media/Backup/Hive/hive-0.5.0-
>
> dev/bin:/usr/local/hadoop/zookeeper-3.3.1/bin:/usr/local/hadoop/hbase-0.20.4/bin:/usr/local/hadoop/nutch-1.1/bin:/opt/hypertable/
> 0.9.3.4/bin:/usr/local/hadoop/hadoop-0.20.2/bin:/home/alex/jetty-
>
> hightide-7.1.5.v20100705/bin:/usr/local/hadoop/jdk1.6.0_20/bin:/usr/local/hadoop/jdk1.6.0_20/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
> PWD=/usr/local/hadoop/hadoop-0.20.2/conf<http://0.9.3.4/bin:/usr/local/hadoop/hadoop-0.20.2/bin:/home/alex/jetty-%0Ahightide-7.1.5.v20100705/bin:/usr/local/hadoop/jdk1.6.0_20/bin:/usr/local/hadoop/jdk1.6.0_20/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games%0APWD=/usr/local/hadoop/hadoop-0.20.2/conf>
> JAVA_HOME=/usr/local/hadoop/jdk1.6.0_20
> ZOOKEEPER_HOME=/usr/local/hadoop/zookeeper-3.3.1
> OLDPWD=/usr/local/hadoop/hadoop-0.20.2
>
>
> ---------------------------------------------------------------------------------------------------------------------------
>
>
> env | grep -i classpath
>
> ---------------------------------------------------------------------------------------------------------------------------
> no output
>
> ---------------------------------------------------------------------------------------------------------------------------
>
>
> All my operations are:
> 1, got source code from http://github.com/kevinweil/hadoop-lzo,compiled
> them successfully,and then
> 2,,copy hadoop-lzo-0.4.4.jar to directory:$HADOOP_HOME/lib of each master
> and
> slave
> 3,Copy all files under directory:../Linux-amd64-64/lib to directory:
> $HADDOOP_HOME/lib/native/Linux-amd64-64 of each master and slave
> 4,and upload a file:test.lzo to HDFS
> 5,then run:hadoop jar $HADOOP_HOME/lib/hadoop-lzo-0.4.4.jar
> com.hadoop.compression.lzo.DistributedLzoIndexer test.lzo to test
>
> is any other configuration needed?
>
> On Sunday, August 01, 2010 11:37:19 am Alex Kozlov wrote:
> > Hi Alex,
> >
> > org.apache.hadoop.io.compress.GzipCodec is in the hadoop-core-*.jar.  You
> > don't need to add /usr/local/hadoop/hadoop-0.20.2/lib/ to the classpath
> > since hadoop shell script does it for you.  What is exactly the command
> you
> > are trying to run?  Can you also give the output of 'env | grep -i
> hadoop'
> > and 'env | grep -i classpath'.
> >
> > Alex K
> >
> > On Sat, Jul 31, 2010 at 1:38 AM, Alex Luya <alexander.luya@gmail.com>
> wrote:
> > > Hello:
> > >    I have followed this link:http://code.google.com/p/hadoop-gpl-
> > >
> > > compression/wiki/FAQ<
> http://code.google.com/p/hadoop-gpl-%0Acompression/w
> > > iki/FAQ>to install lzo compression library,and copy
> hadoop-lzo-0.4.4.jar
> > > to $HADOOP_HOME/lib,and all files under
> > > ..lib/native/Linux-amd64-64 to $HADOOP_HOME/lib/native/Linux-amd64-64,
> > > and run example,but got this errors:
> > >
> > >
> -------------------------------------------------------------------------
> > > --------------------------------------- Exception in thread "main"
> > > java.lang.IllegalArgumentException: Compression codec
> > >
> > >                org.apache.hadoop.io.compress.GzipCodec not found.
> > >
> > >        at
> > >
> > >
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com
> > > pressionCodecFactory.java:96)
> > >
> > >        at
> > >
> > >
> org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionC
> > > odecFactory.java:134)
> > >
> > >        at
> > >
> > > com.hadoop.compression.lzo.LzoIndex.createIndex(LzoIndex.java:202)
> > >
> > >        at
> > >
> > >
> com.hadoop.compression.lzo.LzoIndexer.indexSingleFile(LzoIndexer.java:117
> > > )
> > >
> > >        at
> > >
> > > com.hadoop.compression.lzo.LzoIndexer.indexInternal(LzoIndexer.java:98)
> > >
> > >        at
> com.hadoop.compression.lzo.LzoIndexer.index(LzoIndexer.java:52)
> > >        at
> com.hadoop.compression.lzo.LzoIndexer.main(LzoIndexer.java:137)
> > >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >        at
> > >
> > >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
> > > :39)
> > >
> > >        at
> > >
> > >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> > > mpl.java:25)
> > >
> > >        at java.lang.reflect.Method.invoke(Method.java:597)
> > >        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > >
> > > Caused by: java.lang.ClassNotFoundException:
> > >                org.apache.hadoop.io.compress.GzipCodec
> > >
> > >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > >        at java.security.AccessController.doPrivileged(Native Method)
> > >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > >        at java.lang.Class.forName0(Native Method)
> > >        at java.lang.Class.forName(Class.java:247)
> > >        at
> > >
> > >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:76
> > > 2)
> > >
> > >        at
> > >
> > >
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com
> > > pressionCodecFactory.java:89)
> > >
> > >        ... 11 more
> > >
> > >
> -------------------------------------------------------------------------
> > > ---------------------------------------
> > >
> > > and then I try to add this to hadoop-env.sh:
> > >
> > > export
> > > HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/hadoop/hadoop-0.20.2/lib/
> > >
> > > Same problem as before,this problem is killing me,because it has
> > > surrounded me
> > > for one month.
>

Mime
View raw message